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I.  INTRODUCTION 

A.  AREA  OF  RESEARCH 

The  purpose  of  this  thesis  is  to  propose  a  new  learning  medium  that  takes 
advantage  of  the  latest  developments  in  computer-based  training  (CBT)  and  the  World 
Wide  Web  (WWW)  as  an  innovative  mode  for  delivering  education.  The  research  focus 
is  the  design  of  a  general  framework  for  developing  integrated  Web-based  CBT  tools  that 
combine  hypertext  technology,  computer-based  training  functionalities  and  interactive 
multimedia  to  enhance  learning  effectiveness. 

B.  RESEARCH  QUESTIONS 

•  What  are  the  design  considerations  and  implementation  strategies  for 
combining  hypertext,  multimedia  case  studies  and  group  decision  support 
(GDSS)  software? 

•  What  is  the  ideal  structure  of  a  web  based  training  tool? 

•  What  is  the  best  way  to  implement  a  multimedia  case  study? 

•  How  can  DSS  software  be  used  in  an  interactive  training  tool? 

•  What  are  the  difficulties  associated  with  creating  a  web  based  tool  as  opposed 
to  a  stand-alone  application? 

C.  BENEFITS  OF  STUDY 

There  is  a  need  for  continuous  education  and  training;  most  specifically  training 
in  real-time  for  effective  information  usage.  This  study  contributes  to  meeting  this  need 
through  creation  of  a  framework  that  can  be  used  to  analyze,  design,  implement  and 
evaluate  a  web-based  training  presentation. 


D.  METHODOLOGY 

To  identify  the  required  elements  for  designing  and  developing  a  web-based 
training  program  that  integrates  hypertext,  multimedia  case  studies  and  group  decision 
support  system  (GDSS)  software,  we  conducted  a  survey  and  analysis  of  CBT 
development  issues.  We  also  performed  a  comparison  of  CBT  authoring  tools  to 
determine  which  tool  provided  the  best  features  and  capabilities  for  creating  training 
programs  and  deploying  them  on  the  web. 

To  demonstrate  the  feasibility  of  implementing  an  integrated  training  program  on 
the  Internet,  we  used  the  ELECTRE  1  Model  from  Co-oP  to  develop  a  training  prototype 
using  Macromedia  Director  6  presentation  and  authoring  software.  Co-oP  is  a  Group 
Decision  Support  System  for  Cooperative  Multiple  Criteria  Group  Decision  Making, 
created  by  Dr.  Tung  X.  Bui  of  the  Naval  Postgraduate  School,  Monterey,  California. 

In  order  to  capture  vital  user  feedback  for  the  enhancement  of  the  prototype,  we 
created  an  on-line  evaluation  form  using  NetObjects  Fusion  2.0  and  Microsoft  Access  97. 

E.  RESEARCH  OUTPUT 

A  working  prototype  of  a  Web-based  CBT  was  created  with  Macromedia  Director 
6.0.  The  product  is  available  for  viewing  at  rhttp://www.cimnet.nps.navv.mil/coopl. 

The  software  training  module  is  provided  "as  is."  All  guarantees  either  expressed 
or  implied  are  disclaimed  as  to  the  software  and  its  quality,  performance,  or  fitness  for 
any  particular  purpose. 


F.         ORGANIZATION  OF  THE  STUDY 

Chapter  II  is  a  literature  review  of  CBT.  A  brief  history  of  CBT  is  presented. 
Current  trends  in  CBT  delivery  such  as  multimedia,  authoring  tools  and  Internet/WWW 
platform  deployment  are  discussed. 

Systems  analysis  and  design  procedures  employed  in  developing  the  training 
module  are  discussed  in  Chapter  HI.  The  framework  for  the  development  methodology  is 
derived  from  the  Instructional  Systems  Development  (ISD)  model  discussed  in  Chapter 
II.  Design  considerations  and  trade-off  issues  are  also  discussed. 

Chapter  IV  discusses  the  operating  procedures  and  system  functionality  for  the 
training  module.  This  chapter  includes  a  description  of: 

•  minimum  system  requirements  for  accessing  and  viewing  the  training  module 

•  ideal  system  features  for  optimizing  the  training  module's  access  and  viewing 

•  approximate  download  times  for  various  connection  rates 

Chapter  V  concludes  the  study  by  summarizing  the  research,  discussing  lessons 
learned  and  providing  recommendations  for  future  research. 


II.  LITERATURE  REVIEW 
A.        HISTORICAL  PERSPECTIVE  OF  CBT 

Despite  having  gained  significant  popularity  in  recent  years,  CBT  has  been 
around  for  several  decades.  Maul  and  Spotts  (1993)  trace  CBT's  origins  back  to  the 
1920s  with  Pressey's  invention  of  a  teaching  machine  that  could  "produce  measurable 
amounts  of  learning  in  students."  Much  of  the  credit  for  the  instructional  principles 
behind  CBT  courseware  is  attributed  to  B.F.  Skinner  (Hassett  and  Duke,  1986).  In  the 
1950's  Skinner  developed  a  teaching  machine  which  according  to  Hassett  and  Dukes: 


...presented  instruction  and  questions  on  a  paper  tape  which  the  student 
moved  to  see  a  new  sentences  at  a  time.  The  user  wrote  answers  on  a 
second  tape,  then  uncovered  a  hidden  portion  of  the  teaching  tape  to  see 
whether  the  answer  was  correct. 


Easton  (1997)  defines  CBT  as  "the  use  of  computers  and  multimedia  technology 
for  training  in  a  way  that  promotes  student  interest  and  motivation."  Much  of  the 
literature  reviewed  contained  numerous  acronyms  describing  the  use  of  computers  in 
education  and  training.  Other  CBT  related  terminology  described  by  Bixler  and  Bergman 
(1997)  includes  the  following: 


•  CAI  -  Computer-Assisted  Instruction,  and  CBI  -  Computer-Based 
Instruction;  delivering  instruction  through  the  medium  of  a  computer; 
these  are  alternative  terms  for  computer-based  training 


•  CAL  -  Computer-Assisted  Learning;  using  a  computer  as  an  adjunct 
tool  to  aid  in  a  traditional  learning  situation,  like  classroom  training. 
The  computer  is  a  device  to  assist  the  instructor  during  the  training 
process;  like  a  blackboard  or  handouts 

•  CAT  -  Computer-Assisted  Testing.  Assessing  an  individual  through 
the  medium  of  a  computer.  Individuals  take  the  test  at  the  computer, 
and  the  computer  records  and  scores  the  test.  CAT  is  imbedded  in 
most  computer-based  training 

•  CMI  -  Computer-Managed  Instruction.  The  computer  is  used  to 
register  students,  assign  training  schedules,  generate  and  score  tests, 
and  produce  reports  on  student  progress  and  utilization  of  training 
resources  (Kearsley,  1983) 


Fletcher  and  Rockway  (1986)  date  the  history  of  computer  based  instruction 
(CBI)  as  beginning  in  the  mid- 1950' s.  Much  of  the  initial  CBI  research  and  development 
was  either  performed  by  or  supported  by  the  military.  The  Air  Force  is  credited  with 
being  the  first  military  service  to  begin  developing  CBI  (Fletcher  &  Rockway,  1986).  The 
Air  Force's  Semi-Automatic  Ground  Environment  (SAGE),  was  the  first  "computer- 
based,  on-the-job  training  system"  (Fletcher  &  Rockway,  1986). 

The  Navy  is  regarded  as  the  slowest  military  service  to  begin  participation  in  CBT 
research  and  development  (Fletcher  &  Rockway,  1986).  Despite  its  slow  entry  into  the 
CBT  field,  the  Navy  is  recognized  for  its  success  at  the  U.S.  Naval  Academy,  which  was 
"among  the  first  institutions  of  higher  learning  to  offer  an  entire  course  by  computer" 
(Fletcher  &  Rockway,  1986). 

The  most  widely  recognized  CBT  initiative  involving  Department  of  Defense 
agencies,  was  the  Programmed  Logic  for  Automatic  Teaching  Operations  (PLATO) 
Project.    The    initial    development    period    for   the    PLATO    Project    occurred    from 


approximately  1959  to  1965.  During  this  period,  researchers  received  funding  support 
from  the  Army,  Navy  and  Air  Force.  Fletcher  and  Rockway  (1986)  describe  the  initial 
product,  PLATO  I,  as  "a  single-station  computer-based  teaching  machine  using  the 
ILLIAC  (with  IK  40-bit  words  of  primary  memory,  a  10,000- word  drum,  and  an  add 
time  of  75  microseconds)  as  the  central  processor."  The  PLATO  project  successfully 
provided  the  first  programming  environment  for  computer-based  training. 

Nicholson  (1997)  credits  the  evolution  of  the  microcomputer  in  the  late  1970's  as 
the  springboard  for  the  principle  of  developing  programs  to  provide  electronic  training 
which  supplements  or  replaces  standard  training  manuals.  Programs  developed  during 
this  time  were  consistent  with  one  of  two  fundamental  layouts — linear  programs  and 
branching  programs  (Nicholson,  1997).  The  linear  program  involved  "a  sequence  of  very 
small  learning  steps  punctuated  by  a  test  question."  This  method  provided  immediate 
feedback  to  the  student  as  to  whether  the  question  had  been  answered  correctly.  One  of 
the  most  widely  recognized  linear  programs  is  "Exploring  the  IBM  Personal  Computer". 
The  program  was  distributed  as  an  introductory  disk,  which  was  shipped  with  each  new 
IBM  personal  computer  (Hassett  and  Dukes,  1986).  In  branching  programs,  students  were 
provided  with  one  of  several  instruction  paths  to  help  lead  them  to  correctly  answering 
questions.  The  program  would  not  allow  the  learner  to  continue  to  the  next  step  of  the 
program  without  the  answer  given  being  correct. 

Modern  technological  advances  have  resulted  in  the  creation  of  media-rich 
training  programs,  which  combines  text,  graphics,  animation,  video  and  digital  sound. 
Multimedia  enables  CBT  courseware  developers  to  "create  a  complete  multi-sensory 


learning  program,  allowing  students  to  interact  with  the  material,  and  to  learn  according 
to  their  own  needs,  pace  and  learning  styles."  (Easton,  1997)  These  multi-sensory 
systems  aid  in  the  greater  retention  of  new  knowledge  according  to  Bixler  and  Bergman 
(1997),  who  observed  that  "on  average,  people  remember:  10%  of  what  they  read;  20% 
of  what  they  hear;  30%  of  what  they  see;  and  50%  of  what  they  hear  and  see." 

CBT  program  designs  incorporate  one  or  a  combination  of  the  following 
techniques  (Bixler  &  Bergman,  1997;  Kearsley  1983): 

•  Tutorial  -  useful  for  teaching  factual  information,  simple  discrimination,  rules 
and  simple  application  of  rules 

•  Drill  and  Practice  -  used  after  initial  instruction  to  aid  in  mastery  of  new  skills 
or  information  taught 

•  Training  Games  -  used  as  a  supplement  to  other  instruction.  Training  games 
provide  users  with  motivating  opportunities  to  practice  new  skills  or 
information  taught 

•  Simulation  -  used  primarily  to  practice  skills  in  situations  where  real-world 
application  is  too  expensive  or  dangerous 

•  Problem  Solving  -  helps  student  develop  skills  in  logic,  problem  solving,  and 
following  directions,  and  is  generally  used  to  augment  higher  order  thinking 
skills 

•  Demonstration/Presentation  -  best  used  to  support  the  introduction  of  new 
information 


B. 


CURRENT  TRENDS  IN  CBT  DELIVERY 


1.  Determining  the  Feasibility  of  CBT  Use 

Kearsley  (1983)  stated,  "almost  all  CBT  applications  are  driven  by  the  need  to 
improve  the  efficiency  or  effectiveness  of  a  particular  training  system  or  situation."  This 
assertion  continues  to  hold  true  across  the  wide  spectrum  of  training  and  educational 
organizations.  In  an  era  of  military  and  corporate  downsizing,  training  specialists  are 
continually  searching  for  more  efficient  ways  to  spend  limited  training  dollars.  Allen 
(1996)  originally  highlighted  challenges  that  he  believed  would  have  a  major  impact  on 
training  and  instruction  in  the  early  1980's.  These  challenges  include  the  following: 


The  high  cost  of  energy 

Generally  inefficient  use  of  instructors  due  to  student  overload 

Changing  social  trends 

The  high  cost  of  training  in  multiple  locations 

Too  wide  a  range  of  learner  abilities  and  expectations 

The  increasing  cost  of  downtime  required  by  traditional  methods 

Increased  turn-over  and  worker/student  mobility 

The  inability  of  traditional  training  modes   to  present  instruction 
consistently  across  learner  groups 

General  information  overload 


Many  of  these  same  challenges  are  being  encountered  by  training  and  education 
organizations  today.  These  are  some  of  the  issues  that  form  the  basis  for  exploring  CBT 


as  a  training  option  (Allan  1993;  Rand,  1996;  Gallagher  &  Olbeter,  1997).  Montague 
(1984)  studied  the  use  of  CBT  systems  for  training  in  the  U.S.  Navy  and  identified  the 
following  reasons  for  using  CBT,  although  they  are  not  unique  to  the  U.S.  Navy: 

Provide  students  direct  contact  with  tasks  to  be  learned  via  simulations 

Reduce  the  variability  of  instruction 

Improve  security  and  quality  of  achievement  testing 

Reduce  requirements  for  presence  of  instructors 

Make  training  available  at  anytime  to  match  student  availability 

Provide  training  at  remote  site  where  instructors  are  unavailable 

Reduce  training  time 

Convenience 

Reduce  travel  requirements  and  per  diem  expenses  for  training 

Reduce  development  time  for  teaching  materials  and  revisions 


These  recurring  themes  seem  to  provide  adequate  rationale  for  implementing 
CBT.  However,  CBT  may  not  be  an  appropriate  solution  for  some  training  situations. 
Extreme  care  and  consideration  must  be  exercised  when  determining  when,  where  and 
how  to  use  CBT.  Due  to  the  potentially  high  investment  cost  of  CBT,  organizations  must 
perform  assessments  to  determine:  (1)  the  feasibility  of  incorporating  CBT  into  their 
training  strategy;  (2)  the  type  of  CBT  courseware  that  best  fits  their  training  needs  and  (3) 
the  return  on  investment  or  cost-benefit  analysis  of  CBT.  Kearsley  (1983),  proposed  a 
"CBT  Benefits  Checklist"  (Table  2.1),  to  aid  in  assessing  a  training  situation  and  the 
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potential  benefit  to  be  derived  from  applying  CBT  technology.  The  checklist  is  not 
designed  to  be  a  conclusive  evaluation  tool.  A  majority  of  "yes"  answers  should  be 
viewed  as  a  situation  where  CBT  may  successfully  address  the  organization's  training 
needs  and  warrants  further  investigation.  A  majority  of  "no"  answers  is  indicative  of  a 
situation  where  CBT  would  most  likely  not  fit  the  organization's  training  needs  and 
alternative  methods  should  be  considered. 
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CBT  BENEFITS  CHECKLIST 


Increased  Control 

Are  existing  materials  poorly  used? 

Are  existing  training  programs  taught  inconsistently? 

Is  standardization  of  training  important? 

Is  detailed  tracking  of  learning  needed? 


Yes 


No 


Reduced  Resource  Requirements 

Is  decentralized  (field-based)  training  possible? 

Is  higher  student  throughput  desired? 

Is  a  higher  student-to-instructor  ration  desired? 

Is  expensive  equipment  needed  for  training? 

Individualization 

Is  there  considerable  variation  in  student  backgrounds? 

Is  there  considerable  variation  in  student  abilities? 

Is  there  likely  to  be  considerable  student  variation  in 

terms  of  learning  progress? 

Does  the  instruction  have  to  stand  alone 

(e.g.,  self-study)? 

Timeliness  and  Availability 

Is  it  necessary  to  provide  training  to  many  students 

as  quickly  as  possible? 

Is  it  desirable  to  provide  training  "on  demand" 

i.e.,  whenever  and  wherever  a  student  needs  it? 

Is  there  a  problem  with  students  forgetting  due  to 

premature  training? 

Is  there  a  problem  with  a  shortage  of  qualified 

instructors? 

Reduced  Training  Time 
Would  time  savings  in  training  be  worthwhile? 
Can  the  training  system  and  organization  be 
changed  to  capitalize  on  time  savings? 


(Continued  on  next  page) 
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6.  Improved  Job  Performance 

Is  the  quality  of  job  performance  a  critical  training 

concern? 

Are  there  job  performance  problems  that  improved 

training  can  address? 

7.  Convenience 

Do  employees  already  use  a  computer  system  for 

their  jobs? 

Could  CBT  be  integrated  into  existing  jobs  or 

equipment? 

8.  Change  Agent 

Is  there  a  need  for  new  training  approaches  or 

methods? 

Could  CBT  lead  to  improved  personal  or 

organizational  productivity? 

9.  Increased  Learning  Satisfaction 

Is  the  attrition  or  failure  rate  high? 

Is  there  a  problem  with  student  motivation? 

10.  Reduced  Development  Time 

Is  the  large-scale  development  of  training  materials 
and  program  involved? 

Is  immediate  revision/update  of  training  material 
important? 

Is  instruction  developed  in  terms  of  a  competency- 
based  framework? 


Yes       No 


Table  2.1  CBT  Benefits  Checklist  (Kearsley,  1983) 
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The  purpose  of  a  feasibility  study  is  to  investigate  and  assess  the  degree  of  risk 
associated  with  using  CBT.  According  to  Kearsley  (1983),  there  are  four  major  types  of 
feasibility  that  need  to  be  examined: 

•  Instructional  -  deals  with  the  suitability  of  CBT  for  achieving  the 
instructional  objectives  of  the  training 

•  Organizational  -  focuses  on  social,  policy  or  political  issues  that  may 
affect  the  use  of  CBT 

•  Technical  -  availability  of  CBT  features  or  capabilities  needed 

•  Economic  -  concerns  the  affordability  of  CBT  costs  involved 

Kearsley  further  contends  that  a  CBT  application  should  be  successful  in  all  four 
areas  in  order  to  be  successful. 

Adams  (1993)  offers  a  "decision  aid"  to  help  determine  whether  to  use  CBT  in  a 
corporate  environment.  The  goal  is  to  provide  an  evaluator  with  "a  structured  way  to  look 
at  issues  involving  the  learners  and  the  corporate  climate."  Adams  also  advises  that  the 
aid  should  not  be  viewed  as  a  substitute  for  detailed  cost-benefits  analysis.  The  decision 
aid  should  be  used  as  a  supplement  to  a  cost-benefit  analysis. 

Much  attention  has  been  given  to  measuring  the  return  on  investment  (ROI)  for 
information  technology  acquisitions  in  both  the  public  and  private  sector.  In  assessing  the 
return  on  investment  of  CBT,  Allen  (1996)  examines  the  potential  results  of  CBT  from 
two  main  factors:  cost  savings  and  performance  improvement.  Allen  develops  a  cost 
savings  model  for  CBT  by  illustrating  savings  in  the  following  areas: 
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•  Wages  spent  on  training  and  opportunity  costs 

•  Travel  and  entertainment  costs  for  learners 

•  Travel  and  entertainment  costs  for  trainers 

•  Development  costs 

•  Delivery  system  costs 

In  summarizing  the  research  regarding  CBT's  impact  on  worker  performance, 
Allen  (1996)  concluded  that  "people  learn  better  with  multimedia  training;  they 
remember  what  they  learn  more  accurately  and  longer,  and  they  are  better  able  to  use 
what  they  learn  to  improve  their  performance." 

2.  Cost  Issues 

CBT  programs  generally  come  in  one  of  three  types:  (1)  off-the-shelf;  (2) 
customizable  programs;  and  (3)  programs  built  from  scratch  and  fully  customized  to 
company  specifications  (Bixler  &  Bergman  1997).  Off-the-shelf  training  is  the  cheapest 
and  least  flexible  type  of  CBT  because  the  content  is  fixed  and  cannot  be  changed.  Off- 
the-shelf  CBT  is  considered  appropriate  for  training  situations  where  the  skills  to  be 
taught  are  relatively  generic.  Customizable  CBT  allows  for  the  addition  of  organization 
specific  content  and  is  best  suited  for  teaching  skills  that  are  relatively  unique. 
Customized  training  is  the  most  expensive  type  of  CBT  to  develop  and  implement.  As  a 
result,  this  type  of  CBT  is  recommended  for  situations  where  the  skills  to  be  taught  are 
unique,  there  is  no  existing  CBT  to  teach  the  skills  and  there  is  a  requirement  to  train  a 
large  number  of  people. 


15 


A  variety  of  approaches  have  been  used  for  estimating  the  cost  of  purchasing  or 
developing  CBT.  Bixler  and  Bergman  (1997)  use  a  combination  of  development  and 
delivery  costs.  These  costs  are  dependent  upon  the  customizability  of  the  CBT  program 
and  the  instructional  format  of  the  training.  Tables  2.2  -  2.4  illustrate  the  development 
costs,  delivery  costs  and  budget  matrix  derived  by  Bixler  and  Bergman. 


DEVELOPMENT  COSTS 

Off-the-shelf 

Customizable 

Customized 

Computer 
Program/software 

$100 -$300 

$100 -$5,000 

$100 -$2,000 

Customization  or 
Development 

— 

$1,280 

$30,720  -  $46,080 

Selection  & 
installation 

$640 

$640 

$640 

TOTAL 

$740  -  $940 

$2,020  -  $6,920 

$3 1,460 -$48,720 

Table  2.2  Current  CBT  Development  Costs  (Bixler  &  Bergman,  1997) 
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DELIVERY  COSTS 

Independent  Study 

On-site  Classes 

Off-site  Classes 

Instruction 

— 

$768 

$600  -  $20,000 

Instructional  Support 

$1,280 

$768 

$512 

Selection  &  Enrollment 

— 

— 

$640 

TOTAL 

$1,280 

$1,536 

$1,752 -$21,152 

Table  2.3  Current  CBT  Delivery  Costs  (Bixler  &  Bergman,  1997) 


BUDGET  MATRIX 

Development  Costs 

Delivery  Costs 

Independent 

On-Site 

Off-Site 

Study 

Classes 

Classes 

$1,280 

$1,536 

$1,752- 

$21,152 

Off-the-shelf 

$740- 

$2,020- 

$2,276- 

- 

$940 

$2,220 

$2,476 

Customizable 

$2,020- 

$3,300- 

$3,556- 

- 

$6,920 

$8,200 

$8,456 

Customized 

$31,460- 

$32,740- 

$32,996  - 

- 

$48,720 

$50,000 

$50,256 

Table  2.4  Current  CBT  Budget  Matrix  (Bixler  &  Bergman,  1997) 
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Military  instruction  and  training  requirements  tend  to  be  unique  therefore  most 
CBT  programs  used  within  the  military  must  be  customized.  Cost  figures  for  military 
skills  related  CBT  were  obtained  from  the  Chief  of  Naval  Operations  Office  of  Training 
Technology  (OTT),  paper  on  Interactive  Courseware  (ICW).  Costs  are  estimated 
according  to  the  category  of  ICW  presentation.  The  following  table  provides  a  summary 
of  ICW  presentation  categories  and  estimated  average  cost  per  hour  of  instruction.  (OTT, 
1996) 


Category  Number 

Type  of  Presentation 

Estimated  Avg.  Cost  Per 
Hour  of  Instruction 

1A 

Baseline:  video  with  minor 
text 

$1,901 

IB 

Baseline:  graphics  with 
minor  text 

$2,170 

2 

Medium  simulation 

$3,768 

3 

High  level  simulation 

$7,183 

Table  2.5  Cost  Estimate  for  Interactive  Courseware  (ICW)  (OTT,  1996) 
3.  Hypertext-based  Instruction 

In  researching  instructional  methodologies  and  adult  learning,  Montague  and 
Knirk  (1997)  determined  that  hypertext-based  instruction  is  appropriate  for  conveying 
complex  content.  Traditional  instructional  methods  tend  to  be  linear  in  form  and  more 
suitable  for  well-structured  subject  matter.  Hypertext  provides  for  a  non-sequential, 
random  access  method  of  obtaining  information.  (Montague  &  Knirk,  1997) 

The  structure  of  hypertext,  namely  nodes  and  links,  presents  a  unique  architecture 
for  the  development  of  instructional  materials.  Balsubramanian  (1994)  defined  nodes  as 
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"concepts"  and  links  as  "relationships."  Nodes  can  contain  text,  graphics,  video,  audio, 
animation  and  images.  Links  are  responsible  for  establishing  connectivity  between  nodes; 
thereby  connecting  related  concepts.  The  basic  features  of  a  hypertext  system  are 
(Balasubramanian,  1994): 


•  A  Graphical  User  Interface,  with  the  help  of  browsers  and  overview 
diagrams,  helps  the  user  to  navigate  through  large  amounts  of  information 
by  activating  links  and  reading  the  contents  of  nodes 

•  An  authoring  system  with  tools  to  create  and  manage  nodes  (of  multiple 
media)  and  links 

•  Traditional  information  retrieval  (IR)  mechanisms  such  as  keyword 
searches,  author  searches  etc. 

•  A  hypermedia  engine  to  manage  information  about  nodes  and  links 

•  A  storage  system  which  can  be  a  file  system  or  a  knowledge  base  or  a 
relational  database  management  system  or  an  object-oriented  database 
management  system 


Implementing  a  hypertext  system  on  the  World  Wide  Web  creates  an  instructional 
environment  where  content  can  be  developed,  modified  and  maintained  in  a  timely 
manner.  These  features  are  particularly  attractive  when  there  is  a  need  to  distribute 
current  and  rapidly  changing  information  and  instruction  (Hites  &  Ewing,  1997).  The 
cross-platform  capability  of  the  web  guarantees  users  are  able  to  access  materials  with  a 
variety  of  systems  and  browsers. 
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4.  Multimedia  as  a  Training  Tool 

According  to  the  1996  Computer-Based  Training  Report  (Kemske,  1996),  73%  of 
1,024  survey  respondents  estimated  multimedia  was  used  to  some  degree  in  their 
organization's  courseware.  Respondents  were  also  asked  to  compare  multimedia  to 
traditional  CBT  based  on  five  criteria: 

•  knowledge  outcomes 

•  performance  outcomes 

•  overall  return  on  investment 

•  retention 

•  instruction  completion  time 

The  most  significant  findings  occurred  in  the  area  of  knowledge  outcomes,  where 
79%  of  the  respondents  felt  multimedia  was  better  than  traditional  CBT. 

In  a  1996  survey  of  146  Fortune  1,000  companies  by  TRAINING  and  OmniTech 
Consulting  Group  of  Chicago,  multimedia  as  a  training  delivery  system  accounted  for  an 
estimated  16%  of  the  total  training  hours  delivered  to  employees  (Training,  1996).  For 
the  purposes  of  the  survey,  the  investigators  defined  multi-media  based  training  (MBT) 
as  "...an  interactive  learning  experience  incorporating  the  use  of  either  CD-ROM  or 
World  Wide  Web  technology  (via  the  Internet  or,  more  often,  an  internal  company 
"intranet"  using  the  WWW  protocols)." 

To  further  refine  the  survey  results,  respondents  were  grouped  into  three 
classifications  based  on  the  level  of  multimedia  use  within  their  organization.  The 
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classification  groups  were  light  users,  moderate  users,  and  heavy  users.  Results  for  each 
classification  are  as  follows: 


•  Light  users  -  are  companies  that  deliver  less  than  10  percent  of  total 
training  hours  via  multimedia.  Light  users  comprise  36  percent  of  all 
survey  respondents 

•  Moderate  users  -  deliver  10  percent  to  25  percent  of  total  training 
hours  via  multimedia.  They  comprise  42  percent  of  respondents 

•  Heavy  users  -  comprising  17  percent  of  respondents,  deliver  more 
than  25  percent  of  all  training  hours  via  multimedia 

•  The  remaining  5  percent  of  respondents  declined  to  estimate  a  figure 


In  addition,  the  results  indicated  that  software/basic  computer  skills  remains  the 
most  frequently  taught  subject  via  CBT.  Table  2.6  provides  a  breakdown  of  other  training 
topics  taught  (Training,  1996). 
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SUBJECT 

PERCENTAGE  OF  RESPONDENTS 

Software/Basic  Computer  Skills 

42 

Technical  Training 

25 

Management  Training 

25 

Safety/Regulatory  Compliance 

20 

Product/Industry  Training 

19 

Interpersonal  Skill  Training 

17 

Job  Responsibility  Training 
(Basic,  noncomputer-related) 

15 

Customer  Service  Training 

13 

Sales/Marketing  Training 

13 

Corporate  Culture  Training 

13 

Advanced  Computer  Training 

11 

Table  2.6  Training  Topics  Taught  via  CBT  (Training,  1996) 
5.  Internet-based  Training  (IBT)  and  Web-based  Training  (WBT) 

The  Internet  is  becoming  an  increasingly  popular  alternative  for  delivering 
training  (Appleton,  1997;  McManus  1995;  McGarvey,  1996;  Schneider,  1994).  Training 
delivered  over  the  web  provides  for  easy  distribution  of  knowledge  and  resources 
regionally,  globally,  or  within  an  enterprise.  McManus  (1995)  provides  a  description  of 
the  Internet  as  a  delivery  system  that  also  adequately  describes  the  World  Wide  Web. 
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The  Internet  can  deliver  video,  but  not  as  quickly  as  videotape,  television 
or  CD-ROM.  It  can  carry  real  time  personal  interaction,  but  not  as  well  as 
telephone  or  video  conferencing.  It  can  display  textual  information,  but 
not  as  usefully  as  a  book  or  magazine.  Why  then  should  the  Internet  ever 
be  used.  The  Net  has  two  real  advantages  over  other  media.  It  combines 
advantages  of  other  media  so  that  it  conveys  video  and  sound  better  than  a 
book,  is  more  interactive  than  a  videotape  and,  unlike  CD-ROM,  it  can 
link  people  from  around  the  world  cheaply.  The  second  advantage,  and 
one  that  is  often  overlooked  when  discussing  the  Internet  as  a  delivery 
system,  is  that  it  can  also  be  a  content  provider.  The  Internet  is,  arguably, 
the  largest  and  most  diverse  information  resource  in  the  world  today.  It  is 
possible  to  incorporate  a  wealth  of  information  available  on  the  Net  in 
your  design.  . .  .This  sort  of  immediate  access  to  information  and  resources 
can  not  be  found  with  any  other  medium.  (McManus,  1995) 


In  comparing  WBT  to  traditional  implementations  of  CBT,  Kilby  (1994)  outlines 
the  following  advantages  and  disadvantages  of  WBT: 
Advantages: 


Easy  delivery  of  training  to  users 

Instant  multi-platform  capabilities  (Windows,  Mac,  Unix) 

Easy  updating  of  content 

Shift  from  costly  programming  to  interface  and  content  design 

Quicker  turnaround  of  finished  product 

Requires  less  technical  support 

Billing  options  by  user  ID,  number  of  accesses,  date/time  of  access 

Access  is  controllable 

Options  for  installations  on  private  networks  for  security  or  greater 
bandwidth 

Options  to  link  with  other  training  systems 
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•  Multitasking  capability  suitable  for  electronic  performance  support 
systems  (EPSS) 

•  Vast,  untapped  market  for  training 
Disadvantages: 

•  Limited  formatting  of  content  in  current  browsers 

•  Bandwidth/browser       limitations       may       restrict       instructional 
methodologies 

•  Limited  bandwidth  means  slower  performance  for  sound,  video,  and 
intense  graphics 

•  Someone  must  provide  server  access,  control  usage,  bill  users 

Most  of  the  disadvantages  of  WBT  are  attributable  to  techonolgical  limitations. 
As  new  advancements  in  browser  capabilities  and  solutions  to  bandwidth  limitation 
problems  are  discovered,  WBT's  popularity  as  a  training  alternative,  will  increase. 

6.  Motion  Video 

The  use  of  video  in  CBT  programs  and  on  the  Web  is  becoming  more  common. 
However,  video  requires  enormous  storage  space  and  processing  power  (Joralmon, 
1997).  One  frame  of  video  requires  approximately  one  megabyte  of  storage  space.  Full 
video  uses  30  frames  of  video  per  second  (Joralmon,  1997).  The  Moving  Picture  Experts 
Group  (MPEG)  standard  developed  by  the  International  Standards  Organization  and  the 
International  Electrotechnical  Commission,  is  becoming  a  common  compression 
standard  for  CBT  and  Web  applications  (Joralmon,  1997).  The  three  MPEG  standards 
compress  video  into  files  that  are  smaller  and  more  suitable  for  use  with  current  computer 
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resources.  Specifically,  "MPEG-1  standard  compresses  video  into  files  with  playback 
data  rates  supported  by  current  desktop  computers."  (Joralmon,  1997) 

C.        INSTRUCTIONAL  DESIGN  ISSUES 

Regardless  of  the  type  of  courseware  developed,  CBT,  IBT  or  WBT,  the 
courseware  must  be  based  upon  some  model  of  instruction  and  learning  (Kearsley,  1983; 
McManus,  1995;  Schneider,  1994).  One  of  the  greatest  dangers  associated  with  current 
advancements  in  CBT  development  and  delivery  systems  is  that  the  quality  of  the  courses 
will  decrease  as  responsibility  for  course  development  moves  away  from  the  training  and 
instructional  design  specialist  and  more  towards  the  information  systems  department 
(Fritz,  1993;  Hefner,  1996). 

Instructional  Systems  Development  (ISD)  refers  to  "the  systematic  approach  to 
design  and  development  of  training  materials  and  systems."  (Kearsley,  1983)  The  five 
major  phases  of  ISD  are  shown  in  Figure  2.1.  ISD  procedures  help  to  minimize  the 
problems  most  often  attributed  to  training  program  failures.  It  represents  the  foundation 
for  designing  and  developing  effective  CBT  courseware.  (Kearsley,  1983) 
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IMPLEMENTATION 


EVALUATION 


Figure  2.1  Major  Phases  of  ISD  (Kearsley,  1983) 
McManus  (1996)  uses  Cognitive  Flexibility  Theory  (Spiro,  Feltovich,  Jacobson, 
and  Coulson,  1991)  as  the  foundation  for  advocating  use  of  the  Hypermedia  Design 
Model  (HDM)  for  developing  instruction  for  the  web.  Cognitive  flexibility  theory  focuses 
on  the  nature  of  learning  in  complex  and  ill-structured  domains.  Learning  is  best  achieved 
within  this  environment  through  the  use  of  non-linear  and  multidimensional  exploration 
of  subject  matter  (McManus,  1996).  Hypertext,  by  its  nature,  is  conducive  to  supporting 
the  presentation  of  information  from  multiple  perspectives  to  promote  learning  in 
accordance  with  cognitive  flexibility  theory. 

HDM  (Figure  2.2)  employs  a  "geography/cartography  design  metaphor." 
(McManus,  1996)  A  design  metaphor  is  primarily  concerned  with  how  the  designer 
structures  the  learning  domain  while  creating  the  learning  environment.  Unlike  traditional 
instructional  systems  development,  HDM  focuses  on  learner  objectives-'what  the  learner 
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actually  comes  to  the  environment  wanting  to  learn,"  (McManus,  1996)  and  provides  the 
guidance  to  help  the  learner  meet  those  objectives. 


Define  Your 
Learning  Domain 

Identify  Cases  within 
the  Domain 

i 

' 

Identify  Themes 
&  Perspectives 

< 

' 

Provide  Learner 

Controlled  Navigation 

Through  Cases 

' 

' 

Map  Multiple 
Paths  Linking  Cases 

»            Focus  Learner 
Self-Reflection 

Figure  2.2  Hypermedia  Design  Model  (McManus,  1996) 
Gagne  (1979),  Gery  (1987)  and  Alessi  (1991)  have  formulated  other  systematic 
design  steps.  Easton  (1997)  best  summarizes  the  approach  to  be  taken  when  considering 
developing  CBT  courseware-  "it  must  be  better  thought  out  than  more  conventional 
training  strategies." 

D.        AUTHORING  TOOLS 

The  final  area  that  warrants  discussion  is  that  of  CBT  authoring  tools.  Older  DOS- 
based  authoring  systems  such  as  TenCORE,  Quest  and  Phoenix  Micro  have  lost  a  sizable 
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portion  of  their  market  share  to  newer  Windows-based  systems.  The  proliferation  of 
training  programs  which  incorporate  multimedia;  coupled  with  the  dominance  of 
Microsoft  Windows  in  the  workplace,  has  reshaped  the  CBT  authoring  field  (Fritz,  1993). 
The  current  trend  in  CBT  authoring  software  development  is  to  provide  products  that 
incorporate  tools  for  course  design,  course  delivery,  web  integration  and  course 
administration  features,  in  one  package. 

The  traditional  CBT  authoring  process  involved  an  instructional  designer  who 
knew  what  the  program  should  do  and  a  programmer  who  knew  the  authoring  language. 
The  evolution  of  easy  to  use  authoring  tools  has  enabled  both  instructional  designers  and 
non-instructional  designers  to  create  CBT  courseware  without  the  use  of  a  programmer. 
This  technological  advancement  also  harbors  the  potential  consequence  of  poorly 
designed  courseware  being  produced  by  non-qualified  people.  (Hefner,  1996) 

In  The  1996  CBT  Report  (Kemske,  1996),  survey  respondents  identified  139 
different  CBT  or  multimedia  authoring  products  that  they  either  use  or  have  used 
previously.  According  to  the  report  findings,  the  five  most  commonly  used  authoring 
products  were: 

1.  Authorware 

2.  IconAuthor 

3.  ToolBook 

4.  Quest 

5.  Director 
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The  report  results  disregard  software  version  and  implementation  (DOS  or 
Windows).  AuthorWare,  IconAuthor,  ToolBook  and  Quest  are  considered  dedicated  CBT 
development  products.  Director  is  ordinarily  considered  to  be  a  presentation  developer 
(Kemske,  1996).  Table  2.7  provides  a  comparison  of  the  current  Windows  version  of 
these  products. 


Authorware 

3.5 

IconAuthor 

ToolBook 

11 
Instructor 

QuestNet+ 

Macromedia 

Director  6.0 

Studio 

1  Price 

$4,995 

$1,295 

$1,995 

$3,995 

$995 

Authoring 
Metaphor 

Icon/flowchart 

Icon/flowchart 

Page/object 

Icon/flowchart, 

object  oriented 

authoring 

Icon/object 
oriented  authoring 

Internet 
Capabilities 

Yes 

Yes 

Yes 

Yes 

Yes 

Help 
Wizards 

No 

No 

Yes 

Yes 

No 

Programming 

Language 

Capabilities 

DLLs, 

Xcommands, 

Xtras 

DLLs 

OpenScript 
Java 

ANSIC 

Lingo,  Xtras, 
C++,  Java 

Hypertext 
Capabilities 

Yes 

Yes 

Yes 

Yes 

No* 

Templates 

Yes 

Yes 

Yes 

Yes 

No 

*Hyperlinks  can  be  established  by  using  Lingo  scripting 
Table  2.7  Latest  CBT  Authoring  Tool  Comparison 
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III.  METHODOLOGY 

In  this  project,  the  Instructional  Systems  Development  (ISD)  model,  introduced  in 
Chapter  2,  was  used  as  the  framework  for  the  structured  design  and  development  of  the 
CBT  program.  ISD  is  an  iterative  process  in  that  each  phase  provides  feedback  to  one  or 
more  previous  phases  for  the  purposes  of  modifications  and  enhancements.  The  major 
phases  involved  in  ISD  are: 

•  Analysis 

•  Design 

•  Development 

•  Implementation 

•  Evaluation 

The  purpose  of  this  chapter  is  to  describe  the  process  involved  in  applying  the  ISD 

model  to  the  Electre  training  presentation. 
A.        ANALYSIS 

1.         Determine  Training  Technique 

The  concept  for  the  training  was  extremely  unique  in  that  the  goal  was  to  combine 
hypertext  documents,  multimedia-based  business  case  studies  and  an  on-line  software 
application,  through  seamless  integration.  The  resultant  product  would  be  delivered  over 
the  World  Wide  Web  utilizing  Internet  protocols.  The  multimedia-based  case  study  is  the 
main  portion  of  the  training  program.  The  case  study  is  a  simplified  version  of  an  actual 
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application  of  the  Electre  method  to  solve  a  decision  support  problem.  As  a  result,  the 
demonstration/presentation  technique  for  CBT  program  design  was  selected. 

2.  Determine  Training  Objectives 

The  program  was  designed  to  supplement  classroom  instruction  and/or  textbook 
reading  on  the  Electre  method.  The  desired  outcomes  were  to: 


•  Introduce  the  student  to  the  Electre  method  via  creation  of  a  hypertext 
document  of  the  corresponding  textbook  chapter 

•  Present  the  student  with  an  animated  demonstration  of  the  Electre 
method's  application  to  reinforce  knowledge  gained  from  the  reading 

•  Enhance  student  learning  and  retention  through  a  self-testing  module 

•  Provide  the  student  with  understanding  of  the  underlying  methodology 
of  the  Electre  portion  of  the  Co-oP  decision  support  software  program 

B.         DESIGN 


1.  Script  Design 

The  script  was  created  utilizing  the  facts  contained  in  the  case  study.  Additional 
characters  were  created  in  order  to  develop  a  dialogue  that  would  stimulate  the  user's 
interest.  Careful  consideration  was  given  to  designing  interactions  that  would  not 
intimidate,  frustrate  or  insult  the  user. 

The  teach-back  approach  was  used  to  derive  initial  ideas  for  the  script.  Questions 
and  explanations  arising  from  the  live  presentation  of  the  Electre  method  were  used  in  the 
actual  script.  Beta  testing  was  performed  using  the  first  draft  of  the  script  to  present  the 
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Electre  method  to  several  volunteers.  Volunteer  questions  and  feedback  were  used  to 
modify  the  script's  content.  The  final  script  is  included  as  Appendix  A. 

2.         System  Considerations 

The  minimum  system  configuration  was  one  of  the  first  design  considerations 
discussed.  This  configuration  had  to  be  such  that  a  majority  of  computers  would  be  able 
to  play  the  training  module.  To  this  end  the  following  is  considered  to  be  the  minimum 
system  configuration  to  use  the  training  module: 

•  486  compatible  CPU 

•  At  least  30  Mb  free  hard  disk  space 

•  Super  VGA  video  card  with  a  monitor  capable  of  displaying  640x480 

•  Multimedia  capable  sound  system  with  speakers 

•  Mouse 

•  Internet    Explorer    with    the    Shockwave    plug-in.    (Shockwave    can    be 
downloaded  from  the  Co-oP  Web  site) 

•  Access  to  the  Internet  (recommend  a  28.8  bps  modem  or  higher) 

A  faster  CPU  will  improve  playback  of  the  video.  The  limiting  factor  for  viewing 
this  training  module  over  the  Internet  is  the  speed  of  the  connection,  which  affects  the 
download  time. 
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3.         Media  Selection 

Both  digital  motion  video  and  animation  were  considered  as  media  options  for 
developing  the  case  study  presentation.  Each  format  has  its  own  strengths  and 
weaknesses.  The  evaluation  was  based  on  choosing  the  media  that  would  create  a  quality 
product  given  the  computer  resource  limitations  of  both  the  developer  and  end-user. 

a.         Motion  Video 

Motion  video  will  produce  motion  picture  like  quality  on  a  computer. 
Professional  results  can  be  achieved  with  minimum  investment.  To  get  the  video  from  the 
tape  to  the  computer  requires  a  video  capture  card.  The  average  cost  for  a  card  that 
produces  a  quality  picture  is  somewhere  in  the  $1,000  range. 

The  resolution,  frames-per-second  (fjps),  color  depth  and  duration  (in 
seconds)  are  needed  in  order  to  calculate  storage  requirements  for  full  motion  video.  The 
resulting  value  is  the  storage  requirements  in  bits.  To  convert  this  value  to  bytes,  divide 
by  eight.  The  following  is  an  example  for  uncompressed  video: 

Video  Duration:     5  minutes. 

Resolution:  640  x  480  with  24-bit  color 

Frame  Rate:  16fps 

Compression:         uncompressed 

Storage  space  required  =  (5  (minutes)  x  60  (seconds)  x  640  x  480  x  24  x 
16)/8  (bits  per  byte)  =  4.5  gigabytes  [Using  a  compression  technique  like 
MPEG  or  motion  JPEG  can  reduce  this  to  1 .5  megabits  per  second 

(Andleigh  &  Thakrar,  1996)] 
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Full  motion  video  clips  can  still  be  very  large  even  when  a  compression 
scheme  is  used.  To  play  MPEG  or  motion  JPEG  video  files  on  a  computer  may  require  a 
video  card  that  is  capable  of  decoding  these  file  formats. 

Full  motion  video  provides  fluid  playback  with  professional  results 
however,  this  format  was  not  chosen  for  the  following  reasons: 

•  the  large  file  size  of  the  clips 

•  the  inability  to  control  the  clips  within  the  training  program 

•  the  potential  requirement  for  a  higher  quality  video  card  in  the 
playback  computer  to  decode  the  video  clips 

b.         Animation 

Animation  is  the  colorful  illustration  of  movement  with  rich  sound. 
Animated  imagery,  unlike  video,  can  be  made  machine  readable  (Hapgood,  1997).  As  a 
result,  animation  requires  less  bandwidth  and  storage  space.  Objects  can  be  created  that 
exist  only  in  the  mind  of  the  creator  and  manipulated  in  any  manner. 

Animating  bitmaps  allows  for  greater  developer  control  over  file  sizes. 
There  are  numerous  collections  of  bit-mapped  images  available  commercially  as  well  as 
on  the  Internet.  Images  may  be  selected  from  any  of  these  collections.  The  copyright 
information  describes  license  requirements  and  limitations  for  use  of  the  images. 

The  use  of  a  bit-map  editor  is  required  to  modify  a  single  image  in  order  to 
produce  other  images  that  can  also  be  used  in  a  production.  This  process  of  cutting  and 
pasting  a  single  image  to  produce  several  others  can  be  very  time  consuming.    This  is 
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partly  why  it  takes  approximately  200  man-hours  to  complete  one  hour  of  finished 
multimedia  (Kemske,  1996). 

Animation  file  size  is  directly  related  to  bitmap  resolution,  measured  in 
pixels  per  inch,  and  color  depth.  Resolution  and  color  depth  impact  upon  the  quality  of 
the  bitmap  image.  As  discussed  previously,  animation  requires  less  storage  space  then 
video.  Part  of  the  reason  for  this  is  that  the  bit-mapping  process  can  be  partially  done  by 
the  CPU  of  the  computer  the  images  are  being  shown  on.  Although  this  may  require  less 
storage  space,  it  will  also  require  more  computational  power  from  the  CPU. 

4.  Audio  Considerations 

Sound  is  an  analog  signal  that  is  normally  converted  to  a  digital  signal  for  use  by 
the  computer.  One  way  to  convert  this  signal  is  to  take  many  small  pictures  of  the  analog 
signal's  waveform.  Each  of  these  small  pictures  is  called  a  sample.  Sampling  rate  is  the 
number  of  pictures  that  are  taken  in  a  second.  The  actual  process  of  converting  an  analog 
signal  to  digital  is  much  more  complicated  and  beyond  the  scope  of  this  discussion. 

Sound  quality  is  measured  in  terms  of  the  sampling  rate  at  which  the  sound  is 
recorded.  A  sampling  rate  of  44.1  KHz  indicates  that  44,100  pictures  per  second  are 
taken  of  the  analog  waveform.  A  sampling  rate  of  4  KHz  at  8  bits  is  considered  the 
minimum  acceptable  for  (mono)  voice-grade  sound.  A  sampling  rate  of  8  KHz  at  16  bits 
is  the  minimum  required  for  music.  A  CD  is  sampled  at  44  KHz  at  16  bits. 

The  higher  the  sampling  rate,  the  better  the  sound  quality.  The  number  of  bits 
used  to  store  each  same  also  affects  the  sound  quality.   8  bits  and  16  bits  are  commonly 
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used.  The  more  bits  used,  the  better  the  sound  quality.  The  trade-off  is  that  if  a  higher 
sampling  rate  or  a  higher  number  of  bits  is  used,  the  size  of  the  file  increases 
dramatically. 

5.  Software  Considerations 

Several  software  packages  for  creating  a  CBT  were  considered.  An  authoring 
package,  a  graphics  program,  and  audio  software  were  required  to  create  the  training 
module. 

a.  Authoring  Packages 

In  selecting  an  authoring  system,  consideration  was  given  to  each  of  the 
following: 

•  the  cost  of  the  system 

•  training  that  may  be  required  to  learn  the  system 

•  whether  the  system  provided  the  required  functionality 

There  are  several  types  of  authoring  systems.  Mastering  an  authoring 
system  can  become  very  time  consuming.  A  dedicated  authoring  system  is  the  simplest 
type  of  system.  It  is  designed  for  a  single  user  and  generally  only  allows  for  single 
streams  or  a  single  object.  Timeline-based  authoring  systems  arrange  objects  along  a 
timeline.  The  timeline  is  on  the  screen  or  in  a  window.  The  user  selects  an  object  and 
must  position  it  in  the  timeline.  Structured  multimedia  authoring  is  based  on  object-level 
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construction.  It  is  designed  for  very  complex  multimedia  presentations.  It  allows  for 
modular  construction. 

b.  Graphics  Programs 

The  graphics  program  should  allow  for  the  manipulation  of  different 
image  formats.  The  program's  features  should  allow  the  user  to  cut  and  paste  bit-mapped 
images  and  then  create  new  images  using  this  cut  and  paste  technique.  The  program 
should  have  the  ability  to  save  these  images  to  any  of  the  popular  file  formats. 

Image  conversion  is  also  an  important  consideration.  A  flexible  program 
allows  for  converting  to  different  file  formats  and  changing  the  size  and/or  color  scheme 
of  the  image. 

c.  Audio  Software 

The  software  selected  should  allow  the  user  to  record,  convert,  and  edit 
sound  files.  These  features  are  usually  available  in  one  program.  For  recording,  the  user 
should  have  the  ability  to  select  different  input  sources,  i.e.  microphone,  CD,  and  video. 
A  graphical  display  that  shows  the  waveforms  can  be  used  to  edit  audio  files.  The  ability 
to  cut  and  paste  the  sound  files  is  also  important. 

C.         DEVELOPMENT 

A  timeline-based  authoring  tool  from  Macromedia  called  Director  Suite  was 
selected  for  developing  the  training  module.  The  initial  version  was  5.0  and  was  later 
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upgraded  to  6.0.  Director  was  selected  because  it  has  an  intuitive  interface  and  appeared 
easy  to  learn.  The  software  interface  closely  resembles  a  Hollywood  movie  production,  in 
that  both  contain  cast  members,  scripts,  score  and  stage.  Bundled  with  the  Director  suite 
were  several  other  programs  for  sound  editing,  3-D  rendering  and  some  very  good 
graphics  tools. 

Several  decisions  had  to  be  made  regarding  format  for  the  movie.  Animation 
rather  then  video  was  used  due  to  the  size  of  the  files  related  to  video.  Download  times 
for  the  movie  also  contributed  to  the  choice  of  animation.  Screen  size  was  set  to  640x480 
to  ensure  that  any  user  could  view  the  movie  from  the  Internet  regardless  of  their  screen 
size.  Color  depth  was  set  to  8-bit  to  allow  for  compatibility  with  older  video  displays. 
Audio  settings  were  set  to  a  22KHz  sampling  rate  at  16  bit  to  provide  high  quality  sound. 
These  settings  gave  the  best  sound  quality  while  limiting  required  storage  space. 

1.  Animation  Creation 

As  discussed  previously,  animation  was  chosen  rather  than  motion  video  for  use 
in  the  training  movie  due  to  the  requirement  to  make  the  training  available  over  the 
Internet.  An  exhaustive  search  was  conducted  through  several  collections  of  bit  mapped 
images  to  select  those  images  that  could  be  manipulated  to  obtain  the  necessary 
characters  and  props  for  the  animation. 

Resolution  and  color  depth  were  modified  to  control  file  size.  These  trade-offs 
were  done  for  those  portions  of  the  presentation  where  high  resolution  and  color  depth 
were  not  integral  to  effectively  presenting  the  information.  Careful  consideration  was 
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given  to  balancing  trade-offs  in  order  to  avoid  degrading  the  quality  of  the  overall 
presentation. 

Various  graphics  programs  were  used  to  make  alterations  to  the  images  chosen 
including  Paint  Shop  Pro,  Microsoft  Paint,  Freelance,  Director  XRes,  and  the  Director  6.0 
paint  window.  Paint  Shop  Pro  provided  a  good  all  around  program  allowing  for  importing 
art  from  many  different  types  of  formats,  making  alterations  as  needed,  and  storing  the 
altered  picture  in  a  useful  format,  such  as  a  BITMAP  file  (.bmp),  for  importing  into 
Director  6.0.  The  paint  window  within  Director  6.0  also  provided  a  useful  tool  for  making 
alterations  to  pictures  once  they  were  imported  into  the  movie's  file. 

Using  these  graphics  programs  to  cut  and  paste  the  images  produced  several 
images  different  from  the  original.  These  images  were  then  stored  as  new  images.  This 
was  a  very  time  consuming  process.  The  cutting  and  pasting  of  a  single  image  to  produce 
several  other  images  took  several  hours. 

The  bulk  of  the  animation's  creation  was  accomplished  within  Director  6.0.  The 
score  within  Director  6.0  provided  the  ability  to  put  these  different  images  together 
thereby  giving  the  appearance  of  motion.  Ten  seconds  of  animation  took  approximately  4 
to  6  hours  of  production  time. 

Director  handles  each  object  and  sound  within  the  movie  as  movie  cast  members. 
A  movie  score  is  used  by  Director  to  make  each  cast  member,  both  visual  and  sound, 
appear  at  a  given  time  and  place  within  the  movie.  Control  over  when  and  how  each  cast 
member  enters  and  exits  the  movie  is  also  controlled  in  the  score  with  a  variety  of 
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transition  effects.  Many  different  transitions  were  used  to  break  up  the  flow  of  the  movie 
and  provide  some  variety  for  the  viewer. 

Control  over  the  movies'  flow  and  execution  is  also  provided  in  Director's 
scripting  language  called  Lingo.  Lingo  provides  the  ability  to  control  many  of  the  movies' 
features  such  as  navigation,  play  back  delays,  and  interactivity  commands.  Lingo  was 
used  to  control  navigation,  control  sound  play  back  levels,  and  provide  user  feedback 
within  the  training  movie.  Some  sample  Lingo  script  is  included  in  Appendix  B. 

2.         Audio  Creation 

Audio  in  the  training  module  includes  both  voice  and  music.  The  WAVE  (.wav) 
audio  file  format  was  used  for  recording  due  to  its  universal  use  and  the  flexibility 
provided  within  its  many  optional  sampling  rates.  Various  sampling  rates  were  tried  with 
predictable  results.  Initially,  an  8-bit,  11  KHz  audio  test  recording  was  conducted.  The 
result  was  a  smaller  file  size  with  poor  sound  quality  due  to  a  high  noise  level.  A 
compromise  between  providing  high  quality  sound  and  producing  the  lowest  media  file 
size  was  reached  by  recording  the  sounds  at  a  rate  of  22KHz. 

Two  computer-recording  programs  were  used  for  the  recordings:  Sound  Blaster's 
Wave  Studio  and  Sound  Forge.  Wave  Studio  was  easier  to  use,  but  Sound  Forge  provided 
a  greater  choice  of  sound  editing  and  manipulating  features;  many  of  which  were  not 
needed  for  this  project. 

Individual  recordings  of  each  statement  made  by  the  movie's  participants  were 
made.  This  allowed  for  more  precise  control  and  synchronization  of  cast  members  in  the 
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Director  6.0  score.  Synchronizing  the  player's  voices  with  the  movie  animation  was  a 
time  consuming,  interactive  process.  Introducing  sounds  into  the  movie  in  short  segments 
rather  than  one  long  dialog  facilitated  the  job  of  synchronizing  the  action  with  the  sounds. 
This  does  not  present  a  problem  as  long  as  the  visual  cast  members  are  processed  to 
appear  along  with  the  sound  cast  members.  Delays  were  utilized  within  the  score  to 
synchronize  audio  and  animation.  Synchronization  was  assured  throughout  the  movie  by 
breaking  the  movie  up  into  logical  production  segments. 

Director  6.0  does  not  allow  for  stopping  or  pausing  a  sound  file  during  playback 
and  then  continuing  or  resuming  the  sound  file  from  that  point.  The  animated  portion  of 
the  movie  can  be  paused  and  restarted,  but  the  sound  file  associated  with  that  portion  of 
the  movie  must  restart  at  the  beginning  of  that  sound  cast  member.  This  means  that  in 
order  to  stop  the  movie  and  then  resume  playing  the  movie,  requires  that  both  the 
animated  and  audio  portions  of  a  given  movie  segment  be  replayed. 

3.         Movie  Layout 

The  movie  was  divided  into  two  parts  during  development.  This  was  done  to 
facilitate  movie  production  and  ease  of  Internet  access  and  downloading.  Part  I  introduces 
the  movie  and  case  study.  Part  II  demonstrates  the  ELECTRE  1  method  of  GDSS.  Color 
selection  within  the  entire  movie  was  based  upon  the  developers' preferences. 

Part  I  uses  animated  cartoon  type  graphics  on  a  photo  background  as  well  as  photo 
images  on  an  animated  background  to  demonstrate  the  usefulness  of  both  animated  and 
photo  images.  The  training  was  introduced  in  an  office  environment. 
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Part  II  presents  the  training  material  on  a  simulated  computer  screen  to  represent 
the  use  of  computers  in  GDSS  equipped  conference  rooms.  The  screen  was  divided  into 
four  areas  to  emphasize  particular  areas  of  the  training  materials.  The  four  areas  from  top 
to  bottom  are:  title  area,  working  area,  background  information  area,  and  navigation  area. 

The  title  area  provides  a  title  for  the  given  section  of  the  movie  being  played.  The 
working  area  is  centrally  located  on  the  screen  and  contains  the  critical  information  for 
the  training  topic  under  discussion.  Bolded  text  is  used  to  emphasize  the  current 
information  being  presented  by  the  training  tool.  Transitions  are  also  used  to  draw  the 
viewer's  attention  to  a  particular  part  of  the  screen  and  to  keep  the  viewer  engaged. 

The  background  information  area  is  used  to  provide  the  viewer  with  a  quick 
reference  to  previously  displayed  information  that  is  needed  for  input  to  the  current  topic. 
Bold  text  is  used  to  highlight  pertinent  information  for  the  current  discussion  or 
calculation.  Transitions  are  used  to  focus  the  viewer's  attention. 

Navigation  tools  are  placed  at  the  bottom  of  the  computer  to  facilitate  movie 
viewing  and  navigation.  Navigation  buttons  provided  allow  a  user  to  move  between 
sequential  areas  of  the  movie  and  to  the  Main  Menu  or  Exam  section.  The  user  is  notified 
by  animated  graphics  when  the  current  training  section  has  been  completed  and  it  is  time 
to  move  on  to  the  next  section.  Figure  3.1  is  a  sample  screen  from  Part  II  of  the  movie. 
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Step  1:  Concordance  Matrix 
Development 
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INFORMATION 
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Figure  3.1  Sample  Screen 


D.        IMPLEMENTATION 


The  next  step  in  the  ISD  model  required  that  each  phase's  decisions  up  to  this 
point  be  bought  together  in  a  final  training  product.  The  overarching  goal  for  this  training 
module  was  to  make  it  available  to  users  on  the  Internet.  A  Web-based  integration 
scheme  made  up  of  text-rich  Web  pages  hyperlinked  together  was  chosen  for  the  training 
module.  This  design  scheme  provided  a  logical  base  from  which  to  present  other 
graphical  and  animated  training  media.  The  training  package  was  put  together  using  some 
of  the  most  up-to-date  tools  available  for  Web  site  design  and  implementation. 
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1.  Web  Page  Implementation 

Web  pages  are  made  up  of  HyperText  Markup  Language  (HTML)  and  can  present 
many  different  types  of  media  to  the  user  including  text,  graphics,  sound,  animation,  and 
video.  Most  major  vendors  of  word  processing  software  are  beginning  to  include  the 
capability  to  save  and  convert  files  into  HTML  format.  Desktop  publishing  programs 
such  as  Adobe's  PageMaker  6.5  and  Microsoft's  Word  97  have  this  ability. 

NetObject's  NetObject  Fusion  2.0  was  chosen  to  build  the  site's  Web  pages.  This 
program  provides  the  capability  to  create  an  entire  Web  site  with  automatic  links  between 
each  layer  of  pages.  Objects  can  be  imported  and  placed  within  any  Web  page  desired 
ranging  from  text,  which  can  be  typed  in  many  popular  word  processing  programs,  to 
graphics,  animation,  sound,  and  video.  NetObjects  automatically  creates  the  HTML  code 
as  each  object  or  string  of  text  is  inserted  on  the  Web  page. 

The  creation  of  frames  and  the  placement  of  navigation  buttons  and  text  within 
those  frames  provide  navigation  within  the  Web  site.  This  design  gives  a  visual  clue  to 
the  user  that  other  areas  exist  at  the  Web  site  and  provide  a  ready  means  to  return  to  parts 
of  the  site  previously  visited.  Links  are  also  placed  within  the  main  body  of  each  page  to 
allow  moving  to  definitions  and  training  media  contained  on  other  pages.  NetObjects 
provides  a  quick  method  for  creating  and  designating  each  of  these  types  of  links. 

There  are  also  a  large  number  of  Web  site  layout  schemes  within  NetObjects 
including  buttons,  background  colors,  and  text  formats  and  colors.  Simple,  subdued 
colors  with  a  large  use  of  black  text  on  white  background  were  chosen  to  provide  for  a 
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more  universal  look  and  to  make  the  text  easy  to  read.  Choosing  simple  colors  also  works 
well  with  different  browsers  and  various  display  settings. 

2.         Movie  Internet  Preparation 

Director  6.0  prepares  a  movie  to  play  on  the  Internet  by  compressing  it  and 
changing  its  format  to  a  .dcr  movie.  Director  5.0  used  a  feature  called  Afterburner  to 
"Shock"  the  movie  for  the  Web.  Director  6.0  provides  a  save  option  in  the  file  menu  to 
save  the  movie  as  a  Shockwave  Movie.  An  option  is  also  available  to  create  a  stand  alone 
executable  movie  called  a  projector.  A  great  deal  of  computer  processing  is  required 
during  this  compression  evolution  which  can  be  time  consuming  and  even  overpowering 
when  large  movies  are  processed  on  computers  with  slower  CPUs  or  limited  hard  disk 
space. 

Most  of  the  development  work  was  accomplished  on  a  Pentium  166  computer 
with  a  1.6  gigabyte  hard  disk  and  approximately  60  MB  of  free  space.  The  first  portion  of 
the  training  movie  required  32  MB  of  disk  storage  as  a  Director  movie  but  only  3.4  MB 
as  a  compressed  Shockwave  movie.  Processing  the  compression  on  this  segment  did  not 
cause  any  noticeable  problems  with  the  computer.  However,  the  second  portion  of  the 
training  movie  was  considerably  longer  requiring  nearly  90  MB  of  storage  space  prior  to 
compression. 

A  problem  was  encountered  while  Director  was  compressing  the  second  portion 
of  the  movie.  As  director  goes  through  the  process  of  compression  it  must  first  expand  the 
movie.  In  the  case  of  part  two,  the  movie  expanded  to  158  MB  which  required  more 
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space  than  the  hard  drive  had  available.  It  also  took  a  great  deal  of  time  going  through  the 
process.  The  final  processing  on  this  portion  of  the  film  was  accomplished  on  a  computer 
with  a  Pentium  233  processor  and  4  gigabyte  Ultra-SCSI  hard  drive,  which  significantly 
decreased  the  compression  processing  evolution.  The  hard  drive  had  in  excess  of  500  MB 
of  free  space,  which  supported  the  processing  requirements  of  Director.  The  final  size  of 
the  second  part  after  compression  was  4. 1  MB,  a  significant  reduction  making  it  Internet 
ready. 

E.         EVALUATION 

The  final  phase  of  the  project  development  cycle  involved  the  use  of  a  survey  to 
gain  valuable  feedback  from  the  program  users  regarding  the  training  application.  The 
feedback  areas  of  interest  for  the  development  team  were  the  levels  of  knowledge  gained 
by  the  user,  the  user's  opinion  of  the  Web  site  and  graphics,  and  the  type  of  computer  and 
Internet  connection  equipment  used.  The  NetObjects  Fusion  program  was  used  to  create  a 
Web  interface  with  a  database  to  capture  the  users'  answers  to  the  survey.  A  flat  database 
was  created  with  Microsoft  Access  to  capture  the  inputs  gained  from  the  Web  based 
survey.  Appendix  C  contains  a  printed  version  of  the  Web-based  questionnaire. 
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IV.  OPERATION  OF  THE  GROUP  DECISION  SUPPORT  SYSTEM  TRAINING 

MODULE 


A.  INTRODUCTION 

The  major  emphasis  of  this  thesis  is  to  create  computer-based  training  (CBT)  for  a 
decision  support  system  model  and  to  make  the  CBT  available  for  access  via  the  Internet. 
The  ELECTRE  1  multi-criteria  decision  support  model  and  the  Co-oP  decision  support 
system  application  were  chosen  for  use  to  demonstrate  the  feasibility  of  this  application 
development  as  well  as  to  identify  areas  of  difficulty  and  concern  when  applying  CBT  to 
an  Internet  based  training  (IBT)  environment. 

This  chapter  will  discuss  the  IBT  application,  procedures  for  running  the  IBT,  and 
Internet  browser  requirements. 

B.  INTERNET-BASED  TRAINING  APPLICATION 

The  IBT  application  was  developed  as  one  module  embodied  of  two  parts: 

•  The  first  part  consists  of  web  pages,  which  provide  hyperlinked  text-based 
information  about  numerous  models  used  in  multi-criteria  decision  making.  A 
more  detailed  discussion  is  provided  for  the  ELECTRE  1  model,  which  is  the 
focus  subject  of  the  CBT  module 

•  The  second  part  is  a  case  study  involving  a  decision-maker,  two  assistants  for 
the  decision  making  process,  and  an  actual  business  decision  to  be  made 

The  web  page  documents  provide  an  introduction  to  all  of  the  decision  support 

models  that  are  used  by  the  Co-oP  DSS.    Hyperlinks  for  key  ideas  and  definitions  are 

provided  to  assist  students  in  their  learning  and  understanding.  The  ELECTRE  1  model 

pages  have  been  enhanced  with  excerpts  from  Co-oP:  A  Group  Decision  Support  System 
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for  Cooperative  Multiple  Criteria  Group  Decision  Making  (Bui,  1987),  to  provide  the 
student  with  a  thorough  explanation  of  this  model  prior  to  viewing  the  case  study  portion 
ofthelBT. 

The  second  part  of  the  IBT  is  an  animated,  interactive  training  movie  that  has 
three  sub-lessons.  These  consist  of  an  introduction,  the  ELECTRE  1  model  application 
example,  and  an  exam.  The  presentation  begins  by  providing  a  list  of  course  objectives, 
an  introduction  to  the  decision-makers,  and  the  case  study  details.  Figure  4.1  is  a  screen 
shot  of  the  introduction,  which  provides  the  course  objectives  for  the  ELECTRE  1 
training  module.  These  objectives  provide  the  student  with  the  overall  course  outline, 
which  is  followed  throughout  the  animated  portion  of  the  training.  The  student  will  have 
a  good  understanding  of  the  material  if  each  of  these  points  is  understood,  and  will  be 
able  to  demonstrate  this  knowledge  at  the  end  of  the  training  portion  of  the  module. 
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ELECTRE  Training  Module  Enabling  Objectives 


Understand  the  ELECTRE  Method  as  it  applies  to 
Decision  Support  Systems  to  include: 


Be  able  to  identify  decision  options. 


Be  able  to  define  evaluation  criteria  and  assign 
weights  to  those  criteria. 

Be  able  to  create  a  grading  system  for  evaluation 
the  different  options. 

Be  able  to  understand  and  create  Concordance 
and  Discordance  Matrices. 

Be  able  to  understand  and  apply  the  concept  of 
Thresholds  to  the  Concordance  and  Discordance 
Matrices  to  create  the  Outranking  Matrix. 

Be  able  to  use  the  Outranking  Matrix  to  select  the 
"best"  option. 


Figure  4. 1  Introduction/Course  Objectives 

The  second  sub-lesson  presents  the  ELECTRE  1  model  in  detail  with  illustrations 
of  its  application  in  the  case  study.  The  student  is  also  provided  examples  of  this  case 
study's  application  in  the  Co-oP  GDSS  with  screen  shots  and  explanations  of  how  to  use 
the  Co-oP  program. 

Figure  4.2  is  a  screen  shot  of  the  opening  screen  of  the  ELECTRE  1  training 
section  that  breaks  the  training  down  into  separate  lesson  subjects.  This  menu  allows  the 
student  to  navigate  within  the  training  module  section's  training  subjects,  to  go  to  the 
exam  section,  and  to  exit  the  module  when  complete.  Each  of  the  listed  sections  can  be 
viewed  successively  while  proceeding  through  the  training.  They  can  be  accessed  directly 
from  the  menu,  which  supports  the  review  of  a  particular  section  if  the  student  desires. 
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Figure  4.2  Menu  of  Part  II 

The  third  sub-lesson  tests  the  student's  level  of  understanding  of  key  points 
presented  in  the  training  module  and  provides  for  revisiting  applicable  training  areas 
within  the  D3T.  It  contains  both  theoretical  and  practical  questions  taken  from  portions  of 
the  web  page  training  information  and  the  animated  movie.  Positive  and  negative 
feedback  are  provided  to  guide  the  student  to  the  correct  answer.  The  student  may  also 
return  to  the  training  movie's  ELECTRE  1  sub-lesson  to  review  any  of  the  key  areas 
presented  in  order  to  refresh  and  solidify  his  or  her  knowledge. 
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C.         PROCEDURES  FOR  ACCESSING  AND  RUNNING  THE  IBT 

This  GDSS  training  module  can  be  run  by  accessing  the  Co-oP  training  Web  site, 

which  is  located  at  rhttp://www.cimnet.nps.na vv.mil/coopl.  The  Web  site  training  is 
enhanced  due  to  the  availability  of  Web  pages  containing  textual  instruction  in  the 
various  DSS  models  with  a  focus  on  the  ELECTRE  1  method. 

A  Web  browser  such  as  Netscape  or  Internet  Explorer  must  be  used  to  access  the 
Co-oP  training  site  on  the  Internet.  Web  browser  usage  and  restrictions  for  this 
application  are  discussed  in  the  next  section.  Instructions  regarding  site  content  and 
navigation  are  provided  on  the  Co-oP  Home  Page  with  links  to  Macromedia's  Shockwave 
plug-in  to  enable  the  students  to  download  the  plug-in.  Details  on  obtaining  the  plug-in 
are  also  discussed  in  the  next  section. 

Access  to  the  ELECTRE  1  animated  training  movie  is  provided  from  the 
ELECTRE  1  Web  page.  The  movie  is  built  in  two  parts,  which  was  designed  to  shorten 
the  lengthy  down  load  evolution  for  the  entire  movie.  A  table  listing  approximate  Internet 
download  times  is  provided. 

Part  I  presents  an  introduction  to  the  movie  and  explains  the  navigation  buttons 
used  throughout  the  movie.  Figure  4.3  is  a  screen  shot  of  the  introduction  and  figure  4.4 
is  a  screen  shot  of  the  navigation  scheme  used  throughout  the  movie.  Unlike  other  CBT 
applications,  this  Shockwave  movie  does  not  allow  for  synchronous  playing  of  the 
animation  and  the  audio  portions.  Therefore,  the  stop  button  can  be  used  to  stop  at  any 
given  spot  in  the  movie,  but  the  movie  cannot  be  restarted  from  that  position.  The  current 
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section  being  played  must  be  started  over  in  order  to  provide  for  the  built  in 
synchronization  between  the  animation  and  the  audio. 


COOP  TRAINING  1 


Welcome  to  the  ELECTRE  training  movie! 

This  movie  is  made  up  of  two  parts: 

Part  1 :  Introduction 

Part  2:  ELECTRE  1 
dick  on  the  arrow  below  to  start  Part  1 .  Part  2  will 
x  played  following  the  completion  of  Part  1. 


Figure  4.3  Introduction 
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V  COOP  TRAINING  1 


w 


M 
^ 


Navigating  within  the  Movie: 

Use  the  "Forward"  arrow  to  move  to  the  next  training  sea 
Use  the  "Stop"  button  to  stop  at  any  point  in  the  mo  rie. 
Use  the  "Back"  arrow  to  replay  the  current  movie  se  ction 


Click  on  this 

button  to 

Start  the  Movie 


Figure  4.4  Navigation 
A  link  to  Part  II  of  the  movie  is  provided  within  the  Part  I  section  as  shown  in 
Figure  4.5.  The  student  is  also  provided  a  means  to  exit  the  training  at  any  time  within  the 
movie. 
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v":  COOP  TRAINING  1 


ffi     Click  on  Button  to  Replay  Part  1 


¥\      Click  on  Button  to  Play  Part  2 


[exit] 


Figure  4.5  Play  Part  II 
An  exam  is  provided  for  the  student  following  completion  of  the  ELECTRE  1 
Web  page  and  movie  training.   Figure  4.6  is  a  screen  shot  of  a  sample  question  with  the 
instructions  for  taking  the  exam.  The  picture  also  shows  the  navigation  available  to  return 
to  Part  II  's  Main  Menu  in  order  to  go  to  any  lesson  section  for  review. 
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l-i  COOP  TRAINING  1 

Rracl 

Exercise  Exam 

:  .  ' :.,'  ■ 

Use  the  "Next"  arrow  to  go  to  the  next  question. 
Use  a  mouse  to  select  the  correct  answer. 

Return  to  the  Main  Menu  to  review  a  question's  topic. 

Question       ELECTRE  is  characterized  by;    (source:  excerpt  from  Bui, 
1  of  5          Ltctwe  Notes  m  Computer  Science  -  Co-of) 

A..    Circumventing  the  problem  of  incomplete  comparability 
of  alternatives  through  its  concept  of  outranking. 

B    Deciding  between  non-dominated  alternatives:  e.g., 
option  Al  is  better  than  A2  for  some  criteria  and  A2  is 

better  than  A.1  for  some  other  criteria. 

C.   Ill-structuredness  and  occasional  inconsistency  of  the 

decision  maker's  preferences 
D.   All  of  the  above 

k&.-l 

7/ 

r. jr        ^^^»^B                                                   i*4m    '1--MI  ^v^                          V  \ 

\ 

R0V1« 

Figure  4.6  Exam 
Once  a  student  peruses  the  Web  site  and  interacts  with  the  ELECTRE  1  training 
movie,  a  questionnaire  included  in  Appendix  C  is  provided  to  allow  the  student  to  give 
feedback  on  the  training  site.  This  questionnaire  was  designed  to  obtain  student  input  on 
the  quality  of  the  training,  quality  of  the  Web  site;  opinions  on  the  feasibility  of  Web 
based  training,  and  user's  platform  configuration.  The  results  of  the  survey  are  captured  in 
a  database,  which  can  be  used  to  do  statistical  analysis  on  the  data  provided  and  to 
identify  areas  needing  future  improvements. 

D.        INTERNET  BROWSER  REQUIREMENTS 

The  use  of  multimedia  within  Web  documents  is  becoming  a  major  focal  point  of 
Web  development.  This  is  because  the  Web  is  a  natural  medium  for  the  distribution  of 
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media-based  information  resources  across  computing  platforms.  In  addition,  creating 
Web  ready  media  resources  has  become  simpler  as  development  and  conversion  tools 
continue  to  evolve. 

There  are  a  number  of  things  that  must  be  considered  prior  to  downloading  the 
animated  ELECTRE  1  movie.  They  include  the  type  of  Web  browser  being  used,  the 
computer's  processor,  the  connection  type  in  use  for  the  Internet,  and  the  status  of  the 
hard  disc  drive  on  which  the  Browser  files  are  cached  or  stored. 

Multimedia  materials  referenced  on  Web  documents  cannot  be  played  back  within 
the  Web  browser.  Instead,  when  a  browser  encounters  a  media  format  it  can  not  display,  it 
looks  for  a  helper  application  appropriate  for  the  format.  Newer  Web  browsers  are  now 
being  developed  with  many  of  these  helper  applications  built  in,  however,  adding  media 
viewers  as  plug-ins  can  support  most  media  formats. 

To  view  the  multimedia-animated  movie  provided  with  this  IBT  application,  the 
free  Shockwave  plug-in  from  Macromedia  rhttp;//www.macromedia.com1  must  be 
installed  in  the  student's  Web  browser. 

The  two  most  popular  Web  browsers  in  use  today  are  Netscape  and  Internet 
Explorer.  The  ELECTRE  1  training  movie  has  been  tested  on  both  the  Netscape  3.01  and 
Internet  Explorer  3.0  browsers  with  the  latest  Shockwave  plug-in  installed.  Though  both 
of  these  browsers  support  the  Shockwave  format,  we  have  found  that  the  movie  does  not 
play  properly  on  Netscape  due  to  a  restriction  within  the  Naval  Postgraduate  School's 
Web  server  software.   The  movie  will  play  properly  through  Internet  Explorer  3.0  with 


58 


the  most  recent  Shockwave  plug-in  installed.  This  information  has  been  included  on  the 
appropriate  Web  pages  in  the  training  module. 

Movie  downloading  has  been  tested  on  various  computer  platforms  and  types  of 
connections.  The  computer  platforms  ranged  in  processor  type  from  Pentium  166s  to 
PentiumQ  233s.  Very  little  variance  was  noted  in  the  processing  of  this  movie  between 
the  different  processor  types.  Students  using  these  ranges  of  computers  will  not  notice  any 
significant  difference  in  processing  speed. 

Movie  downloading  was  also  tested  on  various  connection  types  ranging  from 
28.8  Modems  to  T-l  line  connections.  A  significant  difference  in  download  times  was 
expected  and  noted  between  these  various  types.  The  connection  type  rather  than  the 
computer's  processor  is  by  far  the  factor  with  the  greatest  impact  on  the  movie's  download 
time.  Table  4.1  shows  the  approximate  download  times  for  a  variety  of  connection  types. 


Type  of  Connection 

Parti 
3.4  Mb 

Part  2 
4.1Mb 

T-l 

(1.5  Mbps) 

30  sec 

1  min 

LAN 
(10Mbps  shared) 

45  sec 

1  min 

MODEM  14.4  Kbps 

33  min 

45  min 

28.8  Kbps 

15  min 

24  min 

33.6  Kbps 

14  min 

23  min 

56  Kbps 

11  min 

20  min 

Table  4. 1  Download  Times 
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Web  browsers  use  temporary  storage  space  on  the  computer's  hard  disc  drive  to 
store  Web  browser  downloaded  files.  In  the  case  of  Shockwave  movies,  the  hard  disc 
space  is  also  used  for  temporary  storage  while  the  compressed  files  being  received  from 
the  Web  are  processed  in  order  to  play  back  on  the  computer.  Normal  browsing  of  Web 
pages  does  not  have  a  significant  impact  on  the  hard  disk's  storage,  but  multimedia  of  the 
size  of  these  movies,  requires  a  significant  amount  of  additional  hard  disc  space  to  load, 
process  and  run  properly.  It  was  found  that  a  hard  drive  with  less  than  30  MB  of  free 
space  had  a  significant  negative  impact  on  downloading  and  playing  the  entire  ELECTRE 
1  training  movie.  Having  a  minimum  of  30  MB  available  free  space  on  the  browser's  hard 
drive  prior  to  attempting  to  view  the  ELECTRE  1  movie  is  recommended.  This  space 
should  be  available  on  the  drive  in  which  the  browser  stores  its  downloaded  pages,  which 
is  normally  the  C:  drive. 
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V.  SUMMARY  AND  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

A.        SUMMARY  OF  RESEARCH 

This  thesis  was  initiated  to  research  the  feasibility  of  using  various  Internet 
technologies  to  provide  computer-based  training  for  a  Group  Decision  Support  System 
application.  Current  trends  in  Internet  development  and  a  few  of  the  most  up-to-date 
development  tools  were  examined.  This  research  supports  the  idea  of  using  the  Internet  as 
a  viable  means  for  providing  training  material  to  a  broadly  distributed  audience.  It  also 
demonstrates  the  usefulness  of  some  of  the  latest  tools  provided  for  this  type  of 
application  development. 

The  IBT  method  of  distributing  and  executing  training  material  has  some 
advantages  that  can  be  useful  to  the  military.  This  method  provides  the  training  material 
to  an  almost  limitless  audience.  The  only  requirement  for  a  user  to  access  the  material 
would  be  to  have  an  Internet  connection  and  current  browser.  The  training  material's 
controlling  organization  could  make  changes  and  updates  to  the  material  and  almost 
immediately  provide  the  updated  material  for  use  by  the  next  user/student  with  a 
minimum  cost  for  distribution. 

The  limiting  factor  to  full  usefulness  of  the  Internet  for  distributing  training 
materials  is  still  the  bandwidth  available  to  the  user.  Most  modern  CBT  modules  contain 
animation  and/or  video  and  sound  typical  of  the  ELECTRE  1  movie.  Use  of  this  type  of 
media  results  in  large  file  sizes  even  when  the  most  up-to-date  compression  techniques 
are  used.  Downloading  these  training  movies  from  the  Internet  through  a  modem,  which 
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is  the  most  probable  type  of  connection  for  most  users,  takes  a  significant  amount  of  time. 
If  the  computer  training  material  is  distributed  and  used  over  an  organization's  Intranet, 
the  bandwidth  restriction  can  be  overcome  making  the  download  and  processing  time  a 
less  significant  negative  factor. 

A  modification  of  the  ISD  model  for  training  system  development  was  used.  This 
design  methodology  was  found  to  be  straightforward  in  its  application  and  very  useful  to 
our  development  team.  The  following  list  provides  an  estimate  of  the  percentage  of  time 
spent  in  each  phase  of  the  method. 

•  Analysis  10% 

•  Design  15% 

•  Development         60% 

•  Implementation     10% 

•  Evaluation  5% 
B.         LEASONS  LEARNED 

The  tools  available  today  to  create  Internet  training  material  provide  solid  and 
easy  to  use  resources.  As  inexperienced  training  and  Internet  material  producers,  the  tools 
used  to  develop  this  project  were  intuitive  and  well  supported  by  the  software's 
documentation  and  help  function.  There  are  also  a  number  of  good  Internet  resources  and 
"How-to"  books  available  by  professional  users  of  the  tools. 

Using  a  personal  computer  to  develop  multimedia-based  training  applications  for 
use  in  the  Internet  requires  the  latest  hardware.  A  computer  with  a  fast,  high  powered 
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CPU  and  a  minimum  of  32  MB  RAM  (64  MB  RAM  if  possible)  provides  the  required 
horsepower  to  process  movies  in  the  compressed  format  required  for  the  Internet.  Use  of 
a  hard  drive  with  a  high  access  rate  and  a  minimum  of  two  times  the  movie's 
uncompressed  size  in  free  space,  is  also  recommended  to  provide  the  required  processing 
support.  Use  of  a  high  capacity  removable  storage  device,  such  as  a  1  gigabyte  JAZ 
storage  drive,  was  also  found  to  be  very  useful.  The  JAZ  drive  provided  a  back-up  source 
and  allowed  for  the  transportation  of  large  files  to  other  computer  systems. 

The  greatest  challenge  to  this  development  process  was  creating  the  animated  art 
for  the  training  movie.  This  aspect  of  the  development  phase  consumed  the  greatest 
amount  of  time  and  had  the  steepest  learning  curve.  Based  on  the  experience  gained  from 
this  project's  development,  the  time  required  to  create  this  application  again  would  be 
reduced  by  25  to  33%.  To  obtain  a  more  professional  animated  product  will  require 
formal  training  in  animation  development,  or  more  appropriately,  the  services  of  a 
graphic  illustrator. 

C.         RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

The  restriction  of  download  time  may  be  lessened  in  the  future  as  streaming 
technologies  become  maturer.  Streaming  was  attempted  in  this  project  with  unacceptable 
results.  As  this  technology  progresses,  it  should  be  looked  into  as  a  possible  enhancement 
to  Internet  multimedia  training  materials. 

Future  development  should  be  considered  for  other  GDSS  models  and  concepts  as 
a  result  of  the  successful  demonstration  in  this  application  of  the  latest  in  Internet 
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multimedia  development  to  one  aspect  of  the  multi-criteria  decision  models.  Integrating 
Internet  training  with  an  Internet  ready  GDSS  application  would  be  a  good  enhancement 
as  well.  Migrating  the  Co-oP  GDSS  program  to  JAVA-based  applets  would  provide  a 
more  versatile  resource  for  that  type  of  application. 
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APPENDIX  A.  ELECTRE  PRESENTATION  SCRIPT 


Characters:    Narrator 

Mr.  Castella 
Mr.  Henry 
Mrs.  LeFlore 
Applicants  1-5 

Scene  1 

(multimedia  presentation  has  been  launched  by  user  clicking  link  from  text) 

Narrator:  The  following  presentation  is  an  example  of  how  the  Electre  Method  was  applied  to 
an  actual  problem.  The  main  characters  are  Mr.  Castella  (character/obj  appears  on  screen), 
President  of  Castella  Enterprises,  Inc.,  Mr.  Henry,  (character/obj  appears  on  screen)  VP  of 
Operations,  and  Mrs.  LeFlore  (character/obj  appears  on  screen),  head  of  the  Operations 
Research  Department.  Mr.  Castella  is  searching  for  a  replacement  for  his  head  accountant,  who 
is  retiring.  Five  applicants  (add  applicants  A1-A5  to  screen,  one  by  one),  answered  his 
advertisement  placed  in  the  business  section  of  a  local  newspaper. 

Fade  out... 

Narrator:  The  demo  begins  with  Mr.  Henry  and  Mr.  Castella  discussing  how  to  make  a 
selection  from  the  five  applicants. 

Scene  2 

Fade  in...  Mr.  Castella  on  screen  alone  looking  at  applications  (papers,  folders) 
Mr.  Henry  appears  on  screen  from  left  as  if  entering  an  office  and  begins  talking 

Mr.  Henry:  Good  Morning  Mr.  Castella.  I  understand  we  have  received  a  total  of  five 
applicants  for  the  head  accountant  position.  Have  you  selected  the  one  you  wish  to  hire? 

Mr.  Castella:  Good  Morning  Mr.  Henry.  No,  I  have  not  made  a  final  selection  however,  after 
studying  all  of  the  resumes,  I  eliminated  two  of  the  applicants,  (dissolve/fade  out?  A4  and  A5), 
whose  qualifications  were  good,  but  not  precisely  what  I  wanted.  The  three  remaining  applicants 
(reposition  A1-A3  on  screen),  present  a  difficult  choice;  none  of  them  stands  out  from  the 
others  as  clearly  the  best  choice.  I  have  no  idea  how  best  to  decide  among  these  three. 

Mr.  Henry:  Hmrnm.  (Show  Henry  with  thinking  bubble  over  his  head,  then  change  to  light 
bulb  for  idea)  (sounding  very  excited)  I  have  just  the  answer  Mr.  Castella!  During  last  week's 
management  meeting,  Mr.  Andrews  in  Marketing,  briefed  the  group  on  how  his  department 
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applied  a  decision  support  model  called  the  Electre  Method  to  determine  which  campaign 
strategy  was  best  for  next  quarter's  operations. 

Mr.  Castella:  (sounding  doubtful)  Electre  Method  you  say?  Have  you  seen  a  demonstration  of 
this  model?  How  does  it  work?  Do  you  think  it  will  help  me? 

Mr.  Henry:  I  have  not  had  the  opportunity  to  see  the  method  demonstrated.  Mr.  Andrews  was 
so  enthusiastic  about  it  that  I  invited  him  to  provide  a  demonstration  for  the  entire  management 
group  next  month.  He  was  supposed  to  contact  Mrs.  LeFlore  in  Operations  Research  since  she  is 
the  person  who  introduced  his  department  to  the  Electre  Method  and  guided  them  through  the 
steps  involved  in  using  it.  I  think  I  should  contact  Mrs.  LeFlore  and  arrange  for  her  to 
demonstrate  this  Electre  Method  for  you  and  I  using  your  applicant  selection  problem.  When 
would  you  like  me  to  set  the  demonstration  up  for? 

Mr.  Castella:  As  soon  as  possible.  I  have  a  pretty  open  schedule  for  the  next  two  days.  See  if 
you  can  make  this  happen  by  then. 

Mr.  Henry:  Very  well.  I  will  see  what  I  can  do. 

Have  Henry  heading  back  off  screen  form  the  same  direction  he  entered. 

Fade  out... 

Scene  3 
Print  on  screen  "Two  days  later  in  the  Operations  Research  Department" 

Fade  in....  Mrs.  LeFlore  is  on  screen.  Mr.  Castella  and  Mr.  Henry  walk  in  from  the  left. 

Mr.  Henry:  Good  morning  Mrs.  LeFlore  (shaking  Mrs.  LeFlore's  hand).  May  I  introduce  Mr. 
Castella.  He  is  very  anxious  to  see  the  presentation  you  have  prepared  for  us. 

Mrs.  LeFlore:  (smiles  warmly)  Good  Morning  Mr.  Henry.  (Turns  to  Mr.  Castella)  Good 
morning  Mr.  Castella.  It  is  a  pleasure  to  meet  you.  Everything  is  in  place  for  the  presentation 
and  I  am  ready  to  begin  whenever  you  are. 

Mr.  Castella:  (smiles,  shaking  LeFlore's  hand)  Good  morning  and  thank-you  for  preparing 
this  on  such  short  notice.  I  hope  what  you  are  about  to  show  us  will  help  me  decide  which 
applicant  is  best  for  the  company. 

Mrs.  LeFlore:  No  problem  sir.  I'm  sure  we  can  do  something  to  help  you  out.  If  you  gentlemen 
will  follow  me  this  way,  we'll  get  started. 

Fade  out.... 
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Scene  4 
Fade  in...  all  three  characters  are  in  what  appears  to  be  a  briefing  room.  Mr.  Henry  and  Mr. 
Castella  are  sitting  and  Mrs.  LeFlore  is  standing  in  front  of  a  large  presentation  board. 

Mrs.  LeFlore:  The  first  part  of  the  process  involves  data  collection.  The  first  step  in  data 
collection  is  to  define  your  options  (Print  on  screen  "1.  Define  options"  in  upper  left-hand 
corner),  which  represent  all  the  different  possible  actions.  We  should  refer  to  the  three 
remaining  applicants  by  their  real  names.  Mr.  Brown  is  Al  (bring/fade-in  Al  on  screen  but 
print  "Brown"),  Mr.  Smith  is  A2  (bring  A2  on  screen  but  print  "Smith"  )  and  Mr.  Jones  is 
A3  (bring  A3  on  screen  but  print  "Jones").  Do  you  have  any  questions  so  far? 

Mr.  Castella:  No.  Please  proceed. 

Mrs.  LeFlore:  Good.  The  next  step  is  to  define  the  criteria  you  would  like  to  use  for  evaluating 
each  of  the  applicants.  Did  you  bring  the  papers  I  had  faxed  up  to  your  office  yesterday? 

Mr.  Castella:  Yes.  I  have  them  right  here.  I  can't  say  that  I  understand  everything  you  asked 
me  for. 

Mrs.  LeFlore:  (Takes  the  papers  Mr.  Castella  hands  her)  Thank  you.  Don't  worry  sir. 
Things  will  all  come  together  as  we  proceed  through  the  demonstration. 

(Print  on  screen  "2.  Definition  of  criteria"  under  step  1) 

Mrs.  LeFlore:    I  see  you  have  decided  on  five  criteria  Mr.Castella:  Education  (Print  "1. 
Education"  to  screen),  Experience  (Print  "2.  Experience"  to  screen),  Adaptability  (Print 
"3.Adaptability"  to  screen),  Sociability  (Print  "3.  Sociability"  to  screen),  and  Health  (Print 
"4.  Health"  to  screen).  Do  you  still  feel  these  are  valid  criteria  for  evaluating  an  applicant  or 
would  you  like  to  make  some  changes? 

Mr.  Castella:  No,  I  do  not  want  to  make  changes.  I  am  satisfied  with  the  criteria  already 
chosen. 

Mrs.  LeFlore:  O.K.  The  next  step  is  to  assign  weights  to  the  criteria.  (Print  on  screen  "3. 
Weight  of  criteria"  under  step  2)    If  you  will  recall  Mr.  Castella,  the  questionnaire  I  sent  you 
asked  you  to  place  different  levels  of  importance  on  the  five  criteria  with  the  only  condition 
being  that  the  weights  had  to  be  divided  up  out  of  a  total  of  100. 

Mr.  Castella:    Now  here  is  where  I  started  becoming  confused.  I  am  not  sure  that  I  completed 
this  portion  correctly. 

Mr.  LeFlore:  Don't  worry  sir,  we'll  take  a  look  at  what  you've  written  and  see  if  you  want  to 
change  any  of  your  previous  choices.  Let's  see.  According  to  your  answers  here,  Education  is 
assigned  15  (Print  "15"  next  to  "1.  Education").  Experience  is  assigned  15  (Print  "15"  next 
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to  "2.  Experience"  forming  even  column  for  weights).  Adaptability  is  assigned  20  (Print 
"20"  in  column).  Sociability  is  assigned  30  (Print  "30"  in  weight  column).  Health  is  assigned 
20  (Print  "20"  in  weight  column) .  The  weights  you  have  assigned  have  to  be  based  on  the 
level  of  importance  you  feel  each  criteria  holds  with  regard  to  your  decision  making.  Would  you 
say  these  values  correctly  represent  your  assessment? 

Mr.  Castella:  Well,  now  that  you  have  explained  it  that  way,  I  suppose  it  makes  more 
sense... yes  I  am  happy  with  the  values  I  have  assigned. 

(Print  on  screen  "4.  Grading  system  "  under  step  3) 

Mr.  LeFlore:  Very  good  sir.  Now  we  will  help  you  establish  a  grading  system  to  use  for 
evaluating  each  applicant.  In  this  step,  we  will  assign  maximum  numeric  values  corresponding 
to  the  following  alphanumeric  ratings:  Excellent,  Good,  Average,  Fair,  Weak.  After  you  have 
completed  this,  the  result  is  a  table  of  points  to  award  for  each  grade.  Do  you  have  any  questions 
before  we  proceed? 

Mr.  Castella:  No.  Not  at  this  point. 

Mrs.  LeFlore:  Okay  then.  Please  fill  out  this  piece  of  paper  (hands  Castella  a  piece  of  paper), 

with  the  scores  you  would  assign  to  each  criteria  based  on  its  alphanumeric  rating. 

(Show  Castella  looking  over  paper  and  writing  on  it  for  a  brief  period  of  time  before 
handing  it  back  to  LeFlore) 

Mrs.  LeFlore:  Now  if  we  take  the  numbers  you  have  assigned  for  each  criteria  and  rating  (have 
LeFlore  look  as  if  she  is  reading  and  either  inputting  info  into  a  computer  or  writing  on 
presentation  board),  we  come  up  with  the  following  criteria  matrix.  (Print  on  criteria  matrix 
on  screen.  See  following  table: 

Criteria 


1 

2 

3 

4 

5 

Excellent 

10 

10 

10 

20 

15 

Good 

8 

8 

8 

15 

10 

Average 

5 

5 

5 

10 

5 

Fair 

0 

1 

1 

0 

3 

Weak 

0 

0 

0 

0 

1 

Mr.  Castella:  I  think  I  understand  what  we've  done  so  far,  but  where  are  we  headed  next? 

Mrs.  LeFlore:  I'm  glad  you  asked  that  sir.  The  next  step  is  for  you  to  evaluate  each  applicant 
based  on  the  established  criteria.  (Print  to  screen  "5  Evaluation  of  the  applicants"  under  step 
4)  Please  fill  out  this  form  and  base  your  evaluation  on  everything  you  have  learned  about  the 
applicant  through  resumes,  interviews  and  references.  (LeFlore  hands  another  paper  to 
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Castella.  Castella  takes  the  form  and  takes  some  time  fllling  it  out.  You  can  have  him 
consult  some  notes  in  his  briefcase  or  portfolio) 

Mr.  Castella:  (hands  the  paper  back  to  LeFlore)  I  think  this  should  do  it.  How  do  we  use 
these  numbers  to  come  to  a  final  answer? 

Mrs.  LeFlore:  Well  sir,  let's  have  a  look  at  what  you've  written  and  then  we'll  go  over  the 
procedures  for  calculating  the  results.  (Print  on  screen  the  following  matrix: 


Applicant 

Criteria 

1 

2          3 

4 

5 

Brown 

8 

8          5 

15 

10 

Smith 

10 

1          8 

10 

15 

Jones 

5 

10          8 

20 

10 

Mrs.  LeFlore:  This  is  what  the  scores  you've  assigned  for  each  criteria  look  like  in  matrix 
format.  Now  let's  look  at  how  we  calculate  the  results. 

Fade  screen  out.... 

Scene  5 

Print  to  screen  "The  Results" 

Fade  screen  out... 

Print  on  screen  "Step  1  Concordance  Matrix  Development"  in  upper  left-hand  corner  of 
the  screen. 

Mrs.  LeFlore:  (speaking  to  Mr.  Castella)  The  Electre  Method  uses  the  concordance  and 
discordance  coefficients  to  make  paired  comparisons  between  different  options. 

Mr.  Castella:  I'm  afraid  I  don't  quite  follow  you.  What  are  concordance  and  discordance 
coefficients  and  how  are  they  derived? 

Mrs.  LeFlore:  The  Electre  Method  is  based  on  a  rule  written  by  a  French  philosopher,  dating 
back  to  1750.  To  paraphrase,  let's  say  you  have  two  actions,  A&B,  which  you  need  to  choose 
between.  If  action  A  is  better  than  action  B  for  the  majority  of  the  decision  criteria  and  there  is 
no  criteria  for  which  A  is  clearly  worse  than  B,  then  it  stands  to  reason  that  A  outranks  B.  Does 
this  make  sense  to  you  so  far  sir? 

Mr.  Castella:  Yes,  I  understand  what  you  have  said  thus  far,  but  this  does  not  explain  the 
concordance  and  discordance  you  mentioned  earlier. 
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Mrs.  LeFlore:  The  concordance  and  discordance  conditions  are  direct  descendants  of  this 
principle.  The  condition  of  concordance  requires  that  a  majority  of  viewpoints  favor  A.  A 
condition  of  discordance  requires  that  no  criterion  favor  B  excessively. 

Mr.  Castella:  (exclaim  happily)  Now  I  get  what  they  are,  but  how  do  we  use  them? 

Mrs.  LeFlore:  The  Electre  method  uses  the  concordance  and  discordance  coefficients  to  make 
paired  comparisons  between  different  options.  In  our  case,  we  will  use  the  concordance  and 
discordance  coefficients  to  make  paired  comparisons  between  Mr.'s  Brown,  Smith  and  Jones. 
Would  you  like  me  to  go  over  any  of  this  again? 

Mr.  Castella:  No.  I  understand  everything  you've  said  so  far.  What's  next? 

Mrs.  LeFlore:  The  concordance  factor  varies  from  0  to  1.  A  perfect  value  for  the  concordance 
coefficient  is  1 .  The  formula  for  determining  concordance  coefficients  may  be  summarized  as 
follows: 

Print  on  screen  the  following  formula: 


Sum  of  the  weights  of  the  criteria 
by  which  A  outranks  B 
CA/B    =    

Sum  of  the  weights  of  all  criteria 
of  the  model 


Mrs.  LeFlore:  The  concordance  coefficient  indicates  to  what  extent  one  option  is  better  than 
another.  I  will  save  the  discussion  of  the  discordance  coefficient  calculations  for  later  when  we 
are  ready  to  look  at  the  actual  values.  Is  there  anything  you  would  like  to  review  before  we 
move  ahead. 

Mr.  Castella:  No.  Please  continue. 

Mrs.  LeFlore:  Very  well  then.  Utilizing  the  data  in  this  example,  the  Index  of  concordance, 
assuming  Mr  Brown  outranks  Mr.  Smith,  is  calculated  as  follows: 

Print  to  screen  the  "Weight  of  criteria"  scale  and  the  "evaluation  of  applicants"  matrix 

Mrs.  LeFlore:  Performing  a  pair-wise  comparison  of  Mr.  Brown  versus  Mr.  Smith  for  each 
criteria,  we  find  the  following  results.  Mr.  Brown  does  not  outrank  Mr.  Smith  for  criteria  1 
(Highlight  Brown  and  Smith  and  their  values  under  criteria  1  column  in  matrix)  ,  which  is 
education,  so  a  score  of  zero  is  assigned. 
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Print  on  screen  the  following.  Note  that  we  will  add  values  to  this  calculation  as  the 
narrator  proceeds  through  discussion  of  the  calculation 


CBrown/Smith  =     0  + 

100 


Mr.  Castella:  Why  did  you  assign  a  zero? 

Mrs.  LeFlore:  Whenever  option  A  does  not  outrank  option  B  for  the  criteria  being  examined,  a 
score  of  zero  is  assigned. 

Mr.  Castella:  Well  what  happens  when  option  A  does  outrank  option  B?  What  value  do  you 
assign  in  that  case? 

Mrs.  LeFlore:  I'm  glad  you  asked  sir.  As  you  will  see  further  on  in  this  evaluation,  whenever 
option  A  outranks  option  B  for  a  given  criteria,  the  value  assigned  is  the  weight  of  the  criteria. 
Let's  take  a  look  at  criteria  2.  Here  Mr  Brown  outranks  Mr  Smith  (Highlight  Brown  and  Smith 
and  their  values  under  criteria  2  column  in  matrix).  In  this  instance,  the  weight  of  criteria  2, 
which  is  15,  is  added  to  the  formula.  (Print  "15  +"  on  formula  screen).  Does  this  make  sense 
sir? 

Mr.  Castella:  Absolutely. 

Mrs.  LeFlore:  Great.  Why  don't  you  do  the  comparison  for  criteria  3  sir. 

Mr.  Castella:  Okay.  Looking  at  criteria  3,  we  find  that  Mr.  Brown  does  not  outrank  Mr.  Smith, 
(Highlight  Brown  and  Smith  and  their  values  under  criteria  3  column  in  matrix),  therefore 
a  score  of  zero  is  added  (Print  "0  +"  to  formula  screen)  How  did  I  do. 

Mrs.  LeFlore:  Fantastic  sir!  I  think  you  have  the  hang  of  it.  How  about  doing  the  next  criteria? 

Mr.  Castella:  (smiling  and  sounding  very  enthusiastic)  For  criteria  4,  Mr.  Brown  outranks 
Mr.  Smith,  (Highlight  Brown  and  Smith  and  their  values  under  criteria  4  column  in 
matrix),  therefore  a  score  of  30  which  is  the  weight  of  criteria  4,  is  added  to  the  formula.  (Print 
"30  +"  to  formula  screen)  For  criteria  5,  Mr.  Brown  does  not  outrank  Mr.  Smith,  (Highlight 
Brown  and  Smith  and  their  values  under  criteria  5  column  in  matrix),  therefore  a  score  of 
zero  is  added  to  the  formula.  Print  "0"  to  formula  screen. 

Mrs.  LeFlore:  The  outcome  of  the  calculation  is  .45  or  45%.  This  value  is  placed  in  the 
concordance  matrix. 

Print  to  screen  the  following  matrix  which  will  be  filled  in  as  the  instructions  progress: 
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Concordance  Matrix 

Brown       Smith      Jones 
Brown  ~  45 

Smith 

Jones 

Mrs.  LeFlore:  The  same  steps  are  followed  to  determine  the  Index  of  Concordance,  assuming 
that  Mr.  Smith  outranks  Mr.  Brown.  The  results  are  as  follows: 

Print  this  formula  to  screen: 

CSmith/Brown  =     15  +  0  +  20  +  0  +  20 


=  .55 

100 

Mrs.  LeFlore:  This  value  is  added  to  the  matrix  in  the  appropriate  column.  Print  "55"  to 
correct  column  in  matrix 

Concordance  Matrix 

Brown      Smith      Jones 
Brown  ~  45 

Smith  55 

Jones 

Mrs.  LeFlore:  Do  you  have  any  questions  on  what  we've  covered  so  far  sir? 

Mr.  Castella:  No.  I  understand  the  comparison  and  calculations.  Do  we  just  keep  repeating  the 
same  procedure? 

Mrs.  LeFlore:  I'm  glad  this  is  making  sense.  Yes,  the  steps  are  repeated  until  all  of  the 
remaining  pairs  of  alternatives  have  been  evaluated.  In  an  effort  to  conserve  time,  the  results 
have  been  filled  in  for  convenience. 

Fill  in  values  for  matrix  as  follows: 

Concordance  Matrix 
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Brown  Smith  Jones 

Brown  ~  45  35 

Smith  55  -  55 

Jones  85  65 


Mrs.  LeFlore:  We  will  now  move  on  to  the  discordance  coefficient  calculations.  Recall  the 
example  from  our  earlier  discussion  which  required  choosing  between  actions  A  and  B.  Well, 
the  condition  of  discordance  requires  that  no  criterion  favor  option  B  excessively.  The 
discordance  coefficient  indicates  to  what  extent  an  option  contains  discordant  elements  that 
might  make  the  option  unsatisfactory.  Calculation  of  the  discordance  coefficient  requires 
unitizing  the  evaluation  scores,  not  the  weight.  Just  as  in  the  concordance  coefficient 
calculations,  the  discordant  factor's  value  varies  between  0  and  1.  A  'fatal'  score  for  the 
discordance  coefficient  is  1 .  The  formula  for  determining  discordance  coefficients  may  be 
summarized  as  follows: 

Print  the  following  formula  on  the  screen: 

The  greatest  negative  variation 
(i.e.,  B  outranks  A)  between 
the  evaluation  scores  for  a 
single  criterion 


DA/B  =  The  maximum  range  between  the 

highest  possible  score  and 
the  lowest  possible  score 

Mr.  Castella:  This  seems  rather  overwhelming.  Perhaps  if  we  move  on  to  the  calculations 
things  will  become  clearer  for  me. 

Mrs.  LeFlore:  Fine  sir.  Feel  free  to  stop  me  at  any  time  if  you  have  questions  or  would  like  me 
to  go  over  something  again.  Once  again  we  will  perform  pairwise  comparisons.  Let's  begin  by 
assuming  Mr.  Brown  outranks  Mr.  Smith  and  examine  the  evaluation  scores  they  received  for 
criteria  1 .  Print  the  evaluation  scores  matrix  to  the  screen.  Remember,  we  are  looking  for  the 
greatest  negative  variation,  so  for  criteria  1  Mr.  Brown  received  an  8  and  Mr.  Smith  received  a 
10.  Since  we  are  assuming  that  Mr.  Brown  outranks  Mr.  Smith,  he  would  have  to  outrank  him 
by  a  negative  value  of  2.  This  value  is  placed  in  the  numerator.  Print  the  following  to  the 
screen: 


D  Brown/Smith  =  Max  (2, 
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Mr.  Castella :  So  for  criteria  2,  Mr.  Brown  received  an  8  and  Mr.  Smith  received  a  1 .  There  is 
no  negative  value  here  so  what  would  the  answer  be? 

Mrs.  LeFlore:  In  this  instance,  we  would  assign  a  value  of  zero.  How  about  trying  the  next 

one? 

Print  to  screen: 

D  Brown/Smith  =  Max  (2, 0, 


Mr.  Castella:  For  criteria  3,  Mr.  Brown  received  a  5  and  Mr.  Smith  received  an  8.  Therefore  we 
would  place  a  3  in  the  formula  you  have  on  the  board.  Print  to  screen: 

D  Brown/Smith  =  Max  (2, 0, 3, 


Mrs.  LeFlore:  Absolutely.  To  finish  off  the  numerator,  criteria  4  would  be  0  and  criteria  5 
would  be  5.  Print  to  screen: 

D  Brown/Smith  =  Max  (2, 0, 3, 0, 5) 


Mrs.  LeFlore:  Do  you  have  any  questions  before  we  move  on  to  the  set  of  values  in  the 
denominator? 

Mr.  Castella:  No.  Please  continue. 

Mrs.  LeFlore:  Now  for  the  set  of  values  in  the  denominator,  we  use  the  range  between  the 
highest  and  lowest  scores  which  could  have  been  assigned  for  each  criteria.  Recall  our  grading 
criteria  matrix.  Print  to  screen  grading  matrix. 


Criteria 

1 

2 

3 

4 

5 

Excellent 

10 

10 

10 

20 

15 

Good 

8 

8 

8 

15 

10 

Average 

5 

5 

5 

10 

5 

Fair 

0 

1 

1 

0 

3 

Weak 

0 

0 

0 

0 

1 

Mrs.  LeFlore:  Would  you  care  to  take  a  shot  at  filling  in  the  appropriate  values? 
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Mr.  Castella:  Sure.  Criteria  1  is  10,  criteria  2  is  10,  criteria  3  is  10,  criteria  4  is  20  and  criteria  5 
is  14.  How  did  I  do? 

Mrs.  LeFlore:  Perfect.  Our  formula  now  looks  like  this:  Print  to  screen: 

D  Brown/Smith  ■  Max  (2, 0,  3, 0, 5) 

Max  (10, 10, 10, 20, 14) 

Mrs.  LeFlore:  Now  we  have  to  select  the  highest  value  from  each  set.  In  this  case,  the 
numerator  would  be  5  and  the  denominator  is  20.  It  is  important  to  point  out  that  the 
denominator  value  will  be  the  same  for  all  of  our  calculations.  The  discordance  coefficient  is  .25. 
Print  to  screen: 

D  Brown/Smith  =   5/20  =  .25 

Mrs.  LeFlore:  We  create  a  discordance  matrix  and  place  this  value  in  the  appropriate  column. 
Print  to  screen: 

Discordance  Matrix 

Brown         Smith  Jones 

Brown  --  .25 

Smith 

Jones 

Mrs.  LeFlore:  Now  assume  Mr.  Smith  outranks  Mr.  Brown.  What  would  our  calculation  look 
like? 

Mr.  Castella:  Criteria  1  is  0,  criteria  2  is  7,  criteria  3  is  0,  criteria  4  is  5  and  criteria  5  is  0. 
Print  to  screen: 

D  Smith/Brown  =  Max  (0,7,0,5,0) 

Max  (10, 10, 10, 20, 14) 

Mr.  Castella:  The  discordant  coefficient  is  equal  to  7  divided  by  20  or  .35. 
Print  to  screen: 

D  Smith/Brown  =  7/20  =  .35 
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Discordance  Matrix 


Brown 

Smith 

Jones 


Brown 


.35 


Smith 


.25 


Jones 


Mrs.  LeFlore:  Fantastic.  Now  let's  go  ahead  and  fill  in  the  rest  of  the  matrix  to  save  time. 
Print  to  screen: 

Discordance  Matrix 


Brown 


Brown 

-- 

Smith 

.35 

Jones 

.15 

Smith 


.25 


.25 


Jones 

.25 
.50 


76 


Mr.  Castella:  How  do  we  derive  a  final  decision  from  all  of  these  numbers? 

Mrs.  LeFlore:  The  factors  from  the  matrices  are  used  in  conjunction  with  concordance  and 
discordance  thresholds. 

Mr.  Castella:  Where  do  these  thresholds  come  from? 

Mrs.  LeFlore:  They  are  chosen  arbitrarily  by  the  user-  you  in  this  case.  The  concordance 
threshold,  let's  call  is  P,  varies  from  0.5  to  1,  and  is  more  severe  as  it  approaches  1.  The 
discordance  threshold  which  we  will  call  Q,  is  more  severe  as  it  approaches  0. 

Mr.  Castella:  Let's  say  I  decided  to  make  P=.60  and  Q=.30.  How  would  I  use  these  values  to 
come  to  a  decision? 

Mrs.  LeFlore:  We  create  what  is  called  an  Outranking  Matrix  based  on  an  IF-THEN  decision 

table  which  says  the  following: 

Print  to  screen  while  Mrs.  LeFlore  reads  aloud: 


If  Then 

CA/B>=P  and  DA/B<=Q  A  outranks  B 

CB/A>=P  and  DB/A<=Q  B  outranks  A 

A  outranks  B,  and  B  outranks  A  The  options  are  equivalent 

Otherwise  The  comparison  is  characterized  by 

Indetermination  or  incomparability 

Mr.  Castella:  Okay.  Let's  see  how  the  matrix  is  developed. 

Mrs.  LeFlore:  Well  if  we  look  at  the  situation  where  we  assumed  that  Mr.  Brown  outranks  Mr. 
Smith,  the  concordance  coefficient  value  of  .45  is  not  greater  than  or  equal  to  our  P  value  of  .60. 
The  discordance  value  of  .25  is  less  than  the  Q  value  of  .30;  therefore  we  cannot  say  that  Mr. 
Brown  outranks  Mr.  Smith.  A  zero  is  placed  in  the  appropriate  column  of  our  matrix.  Print  to 
screen: 

Brown  Smith  Jones 

Brown  —  0 

Smith 
Jones 

Mr.  Castella:  How  can  we  conclude  this  when  the  Q  value  was  less  than  .30? 
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Mrs.  LeFlore:  Only  one  of  the  two  conditions  in  the  IF  statement  was  met.  In  order  for  the 
THEN  statement  to  be  applicable  all  conditions  in  the  IF  statement  must  be  met. 

Mr.  Castella:  Okay.  That  makes  sense.  Let  me  see  if  I  can  apply  this  to  the  case  of  Mr.  Smith 
outranking  Mr.  Brown.  Now,  the  concordance  coefficient  value  is  .55  which  is  less  than  the  P 
value  and  the  discordance  coefficient  value  is  .35  which  is  greater  than  the  Q  value.  In  this  case, 
the  IF  conditions  are  not  met  and  a  0  is  assigned.  How  was  that? 

Mrs.  LeFlore:  Great!  This  is  what  our  matrix  looks  like  so  far. 
Print  to  screen: 

Jones 


Brown 

Smith 

Brown 

~ 

0 

Smith 

0 

~ 

Jones 

Mrs.  LeFlore:  Let's  try  one  more.  How  about  the  case  where  we  assume  that  Mr.  Brown 
outranks  Mr.  Jones? 

Mr.  Castella:  The  concordance  coefficient  value  is  .35  which  is  less  than  the  P  value,  and  the 
discordance  coefficient  is  .25  which  is  less  than  the  Q  value.  Since  both  conditions  of  the  IF 
statement  have  not  been  met,  a  zero  is  assigned. 
Print  to  screen: 

Brown         Smith  Jones 

Brown  -00 

Smith  0 

Jones 

Mrs.  LeFlore:  Just  for  completeness  sake,  how  about  the  case  where  Mr.  Jones  outranks  Mr. 
Brown? 

Mr.  Castella:  The  concordance  coefficient  is  .85  which  is  greater  than  the  P  value  and  the 
discordance  coefficient  is  .15  which  is  less  than  the  Q  value.  Both  conditions  of  the  IF  statement 
have  been  met.  According  to  the  IF-THEN  Decision  Table,  Mr.  Jones  does  outrank  Mr  Brown. 
What  value  do  we  place  in  the  matrix? 

Mrs.  LeFlore:  We  place  a  1  in  the  matrix.  I  will  go  ahead  and  fill  in  the  rest  of  the  matrix  with 
the  appropriate  values. 
Print  to  screen: 


Brown 

Smith 

Jones 

Brown 

— 

0 

0 

Smith 

0 

— 

0 

Jones 

1 

1 

— 
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Mrs.  LeFlore:  Now  we  are  ready  to  draw  some  final  conclusions.  According  to  our  table,  Mr. 
Jones  outranks  both  Mr.  Brown  and  Mr.  Smith,  the  values  of  1  for  those  conditions  support  a 
condition  of  dominance  on  the  part  of  Mr.  Jones. 
Print  to  screen:  (with  arrows  pointing  from  Jones  to  Brown  and  Smith) 

Brown  Smith 


Jones 

Mr.  Castella:    So,  Mr.  Jones  should  be  selected  for  the  job,  but  what  about  the  relationship 
between  Mr.  Brown  and  Mr.  Smith?  Which  one  is  a  better  choice  between  those  two? 

Mrs.  LeFlore:  I'm  glad  you  asked.  We  ended  up  with  a  case  of  incomparability  between  Mr. 
Brown  and  Mr.  Smith,  meaning  that  we  can  not  prove  any  relationship  between  the  two.  As  a 
result,  we  cannot  draw  any  conclusions  as  to  the  dominance  of  one  over  the  other. 

Mr.  Castella:    I  see.  (Standing  and  walking  over  to  shake  Mrs.  LeFlore's  hand)  This  has 
been  a  great  learning  experience  for  me.  I  see  now  why  Mr.  Andrews  was  so  excited  about  all  of 
this.  Thank  you  for  taking  the  time  to  demonstrate  the  Electre  Method  for  me  today.  I  highly 
endorse  your  teaching  this  to  all  of  our  Department  Managers. 

Mrs.  LeFlore:  Thank  you  for  your  kind  words  sir.  It  has  been  my  pleasure.  I  look  forward  to 
helping  anyone  who  is  interested  in  applying  this  method  for  other  decision  making  situations. 

Mr.  Castella:  I  will  have  Mr.  Henry  send  out  a  memo  to  all  the  Department  Managers.  Thank 
you  again  and  have  a  nice  day. 

Mrs.  LeFlore:  You're  welcome  sir.  Good-bye  (Mr.  Castella  departs) 

Fade  out..... 
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APPENDIX  B.  LINGO  SAMPLES  -  MACROMEDIA'S  PROGRAMMING  SCRIPT 

LANGUAGE 


This  appendix  presents  selected  examples  of  Macromedia's  script  language  Lingo.  Lingo 
was  used  to  control  movie  cast  member  characteristics,  navigation,  interactivity,  and  playback 
timing  within  the  ELECTRE  1  training  movie.  A  helpful  listing  of  Lingo  commands  is  provided 
in  the  Director  6  documentation  and  in  Director  6's  Help  files.  Examples  of  Lingo  script  can  also 
be  found  on  the  Internet  at  the  Macromedia  Web  site  and  links  to  other  sites  provided  there. 

A.  Controlling  Cast  Member  Characteristics 

Sound  Cast  Members: 

on  enter  frame  -  reduces  the  sound  play  back  level  of  a  sound  cast 

set  the  soundLevel  to  1        member  when  the  movie  player  enters  the  frame 
end 

B.  Navigation 

Moving  to  specific  areas  of  the  when  with  user  input  movie: 

on  mouseUp  -  movie  player  goes  to  the  "welcome"  flag  within 

go  to  "welcome"  the  movie  when  the  left  mouse  button  is  "up"  after 

end  being  clicked. 

Movie  navigation  without  user  input: 

on  exitFrame  -  keeps  the  playback  of  the  movie  within  a  frame 

go  to  the  frame  indefinitely  waiting  usually  to  wait  another 

end  command 

Navigation  to  a  different  movie  outside  of  the  current  movie: 

on  mouseUp  -  Goes  to  and  starts  downloading  another  movie 

gotoNetMovie  "http://www.cimnet.nps.navv.mil/coop/brief2/brief2.dcr" 
end 

C.  Interactivity 

Provide  feedback  to  the  user  given  a  specific  input: 

on  mouseUp  -  Provides  the  audible  reply  "clap"  to  the  user  for 

puppetSound  "clap"  the  input  of  mouse  up 

end 


81 


Allow  user  to  control  play  status  of  the  movie: 

on  mouseUp  -  Stops  the  movie  at  the  present  location  waiting 

pause  for  another  input 

end 
D.  Control  Playback  Timing 

Control  synchronization  by  delaying  within  a  frame 

on  enterFrame  -  Stops  the  visual  playback  in  a  given  frame 

delay  6  *  60  while  the  sound  channel  continues  to  play  the 

end  audio  member  (delay  is  designated  for  6  seconds) 
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APPENDIX  C.  Co-oP  TRAINING  SURVEY 

Please  complete  the  following  questionnaire  after  you  have  viewed  the  Decision  Support  System  training 
provided  at  the  http://cimnet.nps.navy.mil/coop  web  site.  In  particular,  we  are  interested  in  your 
feedback  regarding  the  ELECTRE  1  movie.  Completing  the  questionnaire  should  not  take  more  than  15 
minutes.  Thank  you  in  advance  for  your  time  and  interest  in  providing  feedback  for  this  project! 


First  Name Last  Name E-mail 


(Filling  in  the  personal  information  fields  above  is  optional) 

1.  How  much  did  you  know  about  Decision  Support  Systems  (DSS)  prior  to  the  training  session? 

O  A  lot 
O  A  fair  bit 
O  A  little 
O  None 

2.  How  much  did  this  training  session  add  to  your  knowledge  about  DSS? 

O  A  lot 
O  A  fair  bit 
O  A  little 
O  None 

3.  How  much  did  you  know  about  the  ELECTRE  1  model? 

O  A  lot 
O  A  fair  bit 
O  A  little 
O  None 

4.  How  much  did  this  training  session  add  to  your  knowledge  about  the  ELECTRE  1  model? 

O  A  lot 
O  A  fair  bit 
O  A  little 
O  None 
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5.  To  what  degree  do  you  feel  this  training  session  explained  the  concept  of  Concordance? 

O  A  lot 
O  A  fair  bit 
O  A  little 
O  None 

6.  To  what  degree  do  you  feel  this  training  session  explained  the  concept  of  Discordance? 

O  A  lot 
O  A  fair  bit 
O  A  little 
O  None 

7.  To  what  degree  do  you  feel  this  training  session  explained  the  application  of  these  concepts  to  the 
process  of  decision  making? 

O  A  lot 

O  A  fair  bit 

O  A  little 

O  None 

8.  Do  you  feel  you  will  be  able  to  apply  this  DSS  model  to  a  decision-making  requirement  as  a  result  of  this 
training  session? 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 

9.  How  did  you  access  this  training  program? 

O  Local  hard  drive 
O  CD  ROM  drive 
O  WWW  (Internet) 
O  Other 

10.  The  quality  of  the  web  site  pages  is: 

O  Very  good 
O  Good 
O  Average 
O  Bad 
O  Very  bad 
O  No  opinion 
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1 1 .  Did  you  try  to  download  the  Director  Shockwave  movie? 

O  Yes 
O  No 

12.  Could  you  download  it? 

O  Yes 

ONo 

13.  How  long  did  it  take  you  to  down  load  of  the  movies  (Minutes)? 

Parti: Part  2: 

14.  Did  the  movie  play  correctly  on  your  machine? 

O  Yes 

ONo 

15.  What  is  the  speed  of  your  Internet  connection? 

O   14.4  Modem 

O  28.8 

O  36.6 

O  56 

O  128 

OT1 

O  Other 

16.  The  response  time  is: 

O  Very  good 
O  Good 
O  Satisfactory 
O  Unsatisfactory 
O  No  opinion 

17.  Despite  some  time  delay  factor,  demonstration  movies  on  the  Web  are  of  practical  use  for  teaching  and 
learning. 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 

18.  Using  a  Web  document  is  a  good  and  practical  way  to  distribute  teaching  materials. 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 
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19.  As  shown  by  this  courseware,  multimedia  makes  the  learning  process  better. 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 

20.  Both  sequential  access  (following  the  arrows  through  the  tutorial)  and  branching  (via  hyperlinks  and 
buttons)  are  allowed  in  these  documents.  Do  you  think  that  this  is  a  good  way  to  organize  a  courseware? 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 

21.  Hyperlinks  in  the  text  make  it  easier  to  learn. 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 

22.  The  courseware  is  easy  to  learn  and  efficient  to  use. 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 

23.  The  courseware  is  pleasant  to  use. 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 

24.  It  is  easy  to  remember  how  to  use  the  courseware. 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 
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25.  The  courseware  is  interactive  or  potentially  interactive. 

O  Strongly  agree 

O  Agree 

O  Disagree 

O  Strongly  Disagree 

O  No  Opinion 

26.  If  you  are  learning/teaching  or  want  to  learn/teach  DSSs,  will  you  use  this  courseware  or  similar 
documents  on  the  web? 

O  Yes 

ONo 

O  Not  applicable 

27.  The  transitions  within  the  movie  when  played  from  the  Internet  may  not  always  be  smooth.  If  you  use 
this  package  as  a  training  aid  tool,  this  is: 

O  Quite  acceptable 

O  Acceptable 

O  Unacceptable 

O  Definitely  unacceptable 

O  No  opinion 

28.  How  would  you  rate  the  quality  of  the  animations? 

O  Very  good 
OGood 
O  Average 
O  Bad 
O  Very  Bad 
O  No  opinion 

29.  How  would  you  rate  the  quality  of  picture  and  other  artwork? 

O  Very  good 
O  Good 
O  Average 
O  Bad 
O  Very  bad 
O  No  opinion 

30.  Are  you  familiar  with  the  World  Wide  Web  (WWW)? 

O  Very  much 
O  Pretty  much 
O  So  so 
O  Not  really 
O  Not  at  all 
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31.  Are  you  familiar  with  computers? 

O  Very  much 
O  Pretty  much 
O  So  so 
O  Not  really 
O  Not  at  all 

32.  Are  you  familiar  with  multimedia  applications? 

O  Very  much 
O  Pretty  much 
O  So  so 
O  Not  really 
O  Not  at  all 

33.  Are  you  familiar  with  computer  based  training  (CBT)  applications? 

O  Very  much 
O  Pretty  much 
O  So  so 
O  Not  really 
O  Not  at  all 

34.  How  have  you  accessed  and  run  a  multimedia/CBT  training  course  in  the  past  (select  all  that  apply)? 

O  CD-ROM  (Stand-alone  or  LAN  based) 
O  Intranet  application 
O  Internet  application 
O  Not  applicable 

35.  What  kind  of  microprocessor  did  you  use  for  running  this  training  product? 

O  486-Based  Microcomputer 
O  586-Based  Microcomputer 
O  Pentium-Based  Microcomputer 
O  Other 
O  Don't  know 

36.  Please  specify  the  amount  of  system  RAM  in  your  microcomputer? 

O  8  Meg  RAM 

O  16  Meg  RAM 

O  32  Meg  RAM 

O  More  than  32  Meg  RAM 

O  Don't  know 
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37.  For  my  personal  training  needs,  I  prefer  multimedia/CBT  based  training  for  the  following  reasons 
(select  multiple  answers  if  applicable): 

O  It  is  more  effective  than  reading  a  book 

O  It  can  be  replayed  many  times 

O  It  can  be  completed  at  my  own  pace 

O  It  cost  less  than  instructor  lead  training  courses 

O  No  opinion 
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